Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
After a server crash, we can't open our MongoDB instance. This is the status of the service: root@xyz:~# systemctl status mongodsasshard.service ● mongodsasshard.service - MongoDB Database Server Loaded: loaded (/lib/systemd/system/mongodsasshard.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2020-07-14 13:33:11 +03; 2h 53min ago Docs: https://docs.mongodb.org/manual Process: 38452 ExecStart=/usr/bin/mongod --config /etc/mongodsasshard.conf (code=exited, status=14) Main PID: 38452 (code=exited, status=14) Jul 14 13:33:10 xyz systemd[1]: Started MongoDB Database Server. Jul 14 13:33:10 xyz mongod[38452]: 2020-07-14T13:33:10.406+0300 I STORAGE [main] Max cache overflow file size custom option: 0 Jul 14 13:33:11 xyz systemd[1]: mongodsasshard.service: Main process exited, code=exited, status=14/n/a Jul 14 13:33:11 xyz systemd[1]: mongodsasshard.service: Failed with result 'exit-code'. And this is from the end of the log file: 2020-07-14T13:33:10.406+0300 I CONTROL [main] ***** SERVER RESTARTED ***** 2020-07-14T13:33:10.409+0300 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none' 2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] MongoDB starting : pid=38452 port=27022 dbpath=/data/sas/MongoSasShardData 64-bit host=somtr 2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] db version v4.0.12 2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] git version: 5776e3cbf9e7afe86e6b29e22520ffb6766e95d4 2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.1d 10 Sep 2019 2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] allocator: tcmalloc 2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] modules: none 2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] build environment: 2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] distmod: ubuntu1804 2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] distarch: x86_64 2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] target_arch: x86_64 2020-07-14T13:33:10.413+0300 I CONTROL [initandlisten] options: { config: "/etc/mongodsasshard.conf", net: { bindIp: "0.0.0.0", port: 27022 } , replication: { replSetName: "rsSas" }, sharding: { clusterRole: "shardsvr" }, storage: { dbPath: "/data/sas/MongoSasShardData", engine: "wiredTiger", journal: { enabled: true } , wiredTiger: { engineConfig: { cacheSizeGB: 32.0 } } }, systemLog: { destination: "file", logAppend: true, path: "/data/sas/log/mongodsasshard.log" } } 2020-07-14T13:33:10.413+0300 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=32768M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress), 2020-07-14T13:33:11.066+0300 E STORAGE [initandlisten] WiredTiger error (2) [1594722791:65966][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory Raw: [1594722791:65966][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory 2020-07-14T13:33:11.066+0300 E STORAGE [initandlisten] WiredTiger error (-31802) [1594722791:66160][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error Raw: [1594722791:66160][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error 2020-07-14T13:33:11.066+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:66333][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted Raw: [1594722791:66333][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted 2020-07-14T13:33:11.066+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:66388][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory Raw: [1594722791:66388][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory 2020-07-14T13:33:11.066+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:66449][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory Raw: [1594722791:66449][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory 2020-07-14T13:33:11.198+0300 E STORAGE [initandlisten] WiredTiger error (2) [1594722791:198200][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory Raw: [1594722791:198200][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory 2020-07-14T13:33:11.198+0300 E STORAGE [initandlisten] WiredTiger error (-31802) [1594722791:198414][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error Raw: [1594722791:198414][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error 2020-07-14T13:33:11.198+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:198598][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted Raw: [1594722791:198598][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted 2020-07-14T13:33:11.198+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:198663][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory Raw: [1594722791:198663][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory 2020-07-14T13:33:11.198+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:198751][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory Raw: [1594722791:198751][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory 2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (2) [1594722791:211978][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory Raw: [1594722791:211978][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory 2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (-31802) [1594722791:212098][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error Raw: [1594722791:212098][38452:0x7f1f4ceefa40], txn-recover: __wt_txn_recover, 765: Recovery failed: WT_ERROR: non-specific WiredTiger error 2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:212171][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted Raw: [1594722791:212171][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 386: cache server: exiting with 1 pages in memory and 0 pages evicted 2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:212195][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory Raw: [1594722791:212195][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 391: cache server: exiting with 1723 image bytes in memory 2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (0) [1594722791:212215][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory Raw: [1594722791:212215][38452:0x7f1f4ceefa40], connection: __wt_cache_destroy, 395: cache server: exiting with 4059 bytes in memory 2020-07-14T13:33:11.214+0300 W STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version. 2020-07-14T13:33:11.214+0300 W STORAGE [initandlisten] WiredTiger metadata corruption detected 2020-07-14T13:33:11.214+0300 F STORAGE [initandlisten] Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair 2020-07-14T13:33:11.214+0300 F - [initandlisten] Fatal Assertion 50944 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 700 2020-07-14T13:33:11.214+0300 F - [initandlisten] ***aborting after fassert() failure I can't open the server now. I need help. Thank you, Bora
dmitry.agranat commented on Tue, 28 Jul 2020 08:48:18 +0000: Hi, We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket. Regards, Dima dmitry.agranat commented on Wed, 15 Jul 2020 13:55:16 +0000: Hi borayuret@gmail.com, This error indicates a potential file permission issue: 2020-07-14T13:33:11.212+0300 E STORAGE [initandlisten] WiredTiger error (2) [1594722791:211978][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory Raw: [1594722791:211978][38452:0x7f1f4ceefa40], txn-recover: __posix_open_file, 715: /data/sas/MongoSasShardData/journal/WiredTigerLog.0000000104: handle-open: open: No such file or directory Can you check the permission on these files and report back? Thanks, Dima