Info

My server had a 'No space left on device' problem because our disk was running out of inode early today. But after we fixed the disk problem, our mongodb was still unable to start or repair. I desperately need your help : ( And the log information as follow: 2016-07-25T17:29:53.558+0800 I CONTROL [initandlisten] MongoDB starting : pid=10992 port=27017 dbpath=/ick/data2 64-bit host=localhost.localdomain 2016-07-25T17:29:53.558+0800 I CONTROL [initandlisten] db version v3.2.3 2016-07-25T17:29:53.558+0800 I CONTROL [initandlisten] git version: b326ba837cf6f49d65c2f85e1b70f6f31ece7937 2016-07-25T17:29:53.558+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013 2016-07-25T17:29:53.558+0800 I CONTROL [initandlisten] allocator: tcmalloc 2016-07-25T17:29:53.558+0800 I CONTROL [initandlisten] modules: none 2016-07-25T17:29:53.558+0800 I CONTROL [initandlisten] build environment: 2016-07-25T17:29:53.558+0800 I CONTROL [initandlisten] distmod: rhel62 2016-07-25T17:29:53.558+0800 I CONTROL [initandlisten] distarch: x86_64 2016-07-25T17:29:53.558+0800 I CONTROL [initandlisten] target_arch: x86_64 2016-07-25T17:29:53.558+0800 I CONTROL [initandlisten] options: { repair: true, storage: { dbPath: "/ick/data2" } } 2016-07-25T17:29:53.847+0800 I - [initandlisten] Detected data files in /ick/data2 created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2016-07-25T17:29:53.847+0800 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint. 2016-07-25T17:29:53.847+0800 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=18G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2016-07-25T17:29:53.898+0800 E STORAGE [initandlisten] WiredTiger (0) [1469438993:898124][10992:0x7f69edc18d40], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 196608: block header checksum of 675114611 doesn't match expected checksum of 1687733155 2016-07-25T17:29:53.898+0800 E STORAGE [initandlisten] WiredTiger (0) [1469438993:898190][10992:0x7f69edc18d40], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value 2016-07-25T17:29:53.898+0800 E STORAGE [initandlisten] WiredTiger (-31804) [1469438993:898209][10992:0x7f69edc18d40], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic 2016-07-25T17:29:53.898+0800 I - [initandlisten] Fatal Assertion 28558 2016-07-25T17:29:53.898+0800 I - [initandlisten] ***aborting after fassert() failure 2016-07-25T17:29:53.946+0800 F - [initandlisten] Got signal: 6 (Aborted). 0x130e1e2 0x130d119 0x130d922 0x3ee060f7e0 0x3ee0232625 0x3ee0233e05 0x1295b32 0x108b3e3 0x1a57e3c 0x1a57fdd 0x1a583c4 0x199067c 0x1990bf3 0x198dbd7 0x19919de 0x19ae1a8 0x19e2140 0x1a56bfb 0x1a57117 0x1a5726b 0x19f02f8 0x1a54415 0x1a20b7f 0x1a20c5e 0x19df078 0x107322e 0x106f6a3 0xf99268 0x974701 0x97815d 0x3ee021ed5d 0x972279 ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"F0E1E2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F0D119"},{"b":"400000","o":"F0D922"},{"b":"3EE0600000","o":"F7E0"},{"b":"3EE0200000","o":"32625","s":"gsignal"},{"b":"3EE0200000","o":"33E05","s":"abort"},{"b":"400000","o":"E95B32","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C8B3E3"},{"b":"400000","o":"1657E3C","s":"__wt_eventv"},{"b":"400000","o":"1657FDD","s":"__wt_err"},{"b":"400000","o":"16583C4","s":"__wt_panic"},{"b":"400000","o":"159067C","s":"__wt_block_extlist_read"},{"b":"400000","o":"1590BF3","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"158DBD7","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15919DE"},{"b":"400000","o":"15AE1A8","s":"__wt_btree_open"},{"b":"400000","o":"15E2140","s":"__wt_conn_btree_open"},{"b":"400000","o":"1656BFB","s":"__wt_session_get_btree"},{"b":"400000","o":"1657117","s":"__wt_session_get_btree"},{"b":"400000","o":"165726B","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"15F02F8","s":"__wt_curfile_open"},{"b":"400000","o":"1654415"},{"b":"400000","o":"1620B7F","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"1620C5E","s":"__wt_metadata_cursor"},{"b":"400000","o":"15DF078","s":"wiredtiger_open"},{"b":"400000","o":"C7322E","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C6F6A3"},{"b":"400000","o":"B99268","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"574701","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"57815D","s":"main"},{"b":"3EE0200000","o":"1ED5D","s":"__libc_start_main"},{"b":"400000","o":"572279"}],"processInfo":{ "mongodbVersion" : "3.2.3", "gitVersion" : "b326ba837cf6f49d65c2f85e1b70f6f31ece7937", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-573.22.1.el6.x86_64", "version" : "#1 SMP Wed Mar 23 03:35:39 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "B389239A9108B604FCA797D0F968E253E7BF9F5E" }, { "b" : "7FFEC8C76000", "elfType" : 3, "buildId" : "2F78F7B8A7307DD9C340F3CA735BE2CAA9C157D1" }, { "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "B84C31B86733DE212F6886FE6F55630FE56180A9" }, { "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "A30A68D2F579614CBEA988BDAAC20CD56D8C48FC" }, { "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "95159178F1A4A3DBDC7819FBEA2C80E5FCDD6BAC" }, { "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "29B61382141595ECBA6576232E44F2310C3AAB72" }, { "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "5D42D8781F5FF4C06B4F839B85B21786AA44488D" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "989FE3A42CA8CEBDCC185A743896F23A0CF537ED" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9A6E4BDFA184364D81F7DFD789474C3FB8F98A00" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "C56DD1B811FC0D9263248EBB308C73FCBCD80FC1" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "8E6FA4C4B0594C355C1B90C1D49990368C81A040" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "959C5E10A47EE8A633E7681B64B4B9F74E242ED5" }, { "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "441FA45097A11508E50D55A3D1FF169BF2BE7C62" }, { "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "F62622218875795666E08B92D176A50791183EEC" }, { "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "152E2C18A7A2145021A8A879A01A82EE134E3946" }, { "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "B8DEDADC140347276164C729418C7A37B7224135" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "4BDFC7A19C1F328EB4FCFBCE7A1E27606928610D" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "C39D7FFB49DFB1B55AD09D1D711AD802123F6623" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "E6798A06BEE17CF102BBA44FD512FF8B805CEAF1" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x130e1e2] mongod(+0xF0D119) [0x130d119] mongod(+0xF0D922) [0x130d922] libpthread.so.0(+0xF7E0) [0x3ee060f7e0] libc.so.6(gsignal+0x35) [0x3ee0232625] libc.so.6(abort+0x175) [0x3ee0233e05] mongod(_ZN5mongo13fassertFailedEi+0x82) [0x1295b32] mongod(+0xC8B3E3) [0x108b3e3] mongod(__wt_eventv+0x40C) [0x1a57e3c] mongod(__wt_err+0x8D) [0x1a57fdd] mongod(__wt_panic+0x24) [0x1a583c4] mongod(__wt_block_extlist_read+0x6C) [0x199067c] mongod(__wt_block_extlist_read_avail+0x33) [0x1990bf3] mongod(__wt_block_checkpoint_load+0x3B7) [0x198dbd7] mongod(+0x15919DE) [0x19919de] mongod(__wt_btree_open+0xC68) [0x19ae1a8] mongod(__wt_conn_btree_open+0x140) [0x19e2140] mongod(__wt_session_get_btree+0xEB) [0x1a56bfb] mongod(__wt_session_get_btree+0x607) [0x1a57117] mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a5726b] mongod(__wt_curfile_open+0x1E8) [0x19f02f8] mongod(+0x1654415) [0x1a54415] mongod(__wt_metadata_cursor_open+0x5F) [0x1a20b7f] mongod(__wt_metadata_cursor+0x7E) [0x1a20c5e] mongod(wiredtiger_open+0x15F8) [0x19df078] mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x45E) [0x107322e] mongod(+0xC6F6A3) [0x106f6a3] mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xf99268] mongod(_ZN5mongo13initAndListenEi+0x321) [0x974701] mongod(main+0x15D) [0x97815d] libc.so.6(__libc_start_main+0xFD) [0x3ee021ed5d] mongod(+0x572279) [0x972279] ----- END BACKTRACE -----

Top User Comments

thomas.schubert commented on Sat, 19 Nov 2016 15:32:05 +0000: Hi Paddyzhu, The likely cause of this issue was fixed in WT-2897, and the fix is included in MongoDB 3.2.10 and later. Since this issue has been resolved, I am closing this ticket as a duplicate. Kind regards, Thomas ramon.fernandez commented on Mon, 3 Oct 2016 16:48:56 +0000: Apologies for the radio silence Paddyzhu. The goal behind looking at your data files was to see if this issue had corrupted them, or if only the repaired files were affected. If you haven't experienced any further issues and your data integrity hasn't been affected (you can run validate(), but note it's resource-intensive) then we don't need your files. Can you please confirm that you're not experiencing any issues that could be due to data corruption? If you are, then you should upload your database files. Thanks, Ramón. paddyzhu commented on Fri, 23 Sep 2016 01:29:34 +0000: Hi everyone, We replaced our files in /usr/local/mongodb/data with your repair_attempt.zip and the issue is fixed! Thank you so much. And since it is done do you still want our contents of /usr/local/mongodb/data ? If so I will upload it for you : ) paddyzhu commented on Fri, 23 Sep 2016 00:44:50 +0000: Thank you! But It seems like I might just post the wrong log since I mixed other's with my own one. please check the following one and our mongodb.log I uploaded 2016-09-22T15:34:24.570+0800 I CONTROL [initandlisten] MongoDB starting : pid=1885 port=27017 dbpath=/usr/local/mongodb/da ta 64-bit host=iZ625ztxpiuZ 2016-09-22T15:34:24.570+0800 I CONTROL [initandlisten] db version v3.2.7 2016-09-22T15:34:24.570+0800 I CONTROL [initandlisten] git version: 4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2 2016-09-22T15:34:24.570+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013 2016-09-22T15:34:24.570+0800 I CONTROL [initandlisten] allocator: tcmalloc 2016-09-22T15:34:24.570+0800 I CONTROL [initandlisten] modules: none 2016-09-22T15:34:24.570+0800 I CONTROL [initandlisten] build environment: 2016-09-22T15:34:24.570+0800 I CONTROL [initandlisten] distmod: rhel62 2016-09-22T15:34:24.570+0800 I CONTROL [initandlisten] distarch: x86_64 2016-09-22T15:34:24.570+0800 I CONTROL [initandlisten] target_arch: x86_64 2016-09-22T15:34:24.570+0800 I CONTROL [initandlisten] options: { repair: true, storage: { dbPath: "/usr/local/mongodb/dat a", engine: "wiredTiger" } } 2016-09-22T15:34:24.593+0800 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint. 2016-09-22T15:34:24.593+0800 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=4G,sessio n_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compre ssor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2016-09-22T15:34:24.615+0800 E STORAGE [initandlisten] WiredTiger (0) [1474529664:615769][1885:0x7fd02f352d40], file:Wired Tiger.wt, connection: read checksum error for 4096B block at offset 12288: block header checksum of 1717798939 doesn't matc h expected checksum of 3759426581 2016-09-22T15:34:24.615+0800 E STORAGE [initandlisten] WiredTiger (0) [1474529664:615802][1885:0x7fd02f352d40], file:Wired Tiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value 2016-09-22T15:34:24.615+0800 E STORAGE [initandlisten] WiredTiger (-31804) [1474529664:615813][1885:0x7fd02f352d40], file: WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic 2016-09-22T15:34:24.615+0800 I - [initandlisten] Fatal Assertion 28558 2016-09-22T15:34:24.615+0800 I - [initandlisten] ***aborting after fassert() failure 2016-09-22T15:34:24.635+0800 F - [initandlisten] Got signal: 6 (Aborted). 0x1337cb2 0x1336be9 0x13373f2 0x393e60f7e0 0x393e2325e5 0x393e233dc5 0x12bea52 0x10af8d3 0x1a9f62c 0x1a9faed 0x1a9fed4 0x1 9d18fc 0x19d1e73 0x19cee15 0x19d2eb9 0x19ef92c 0x1a257c0 0x1a9e357 0x1a9e899 0x1a9e9bb 0x1a33db8 0x1a9b515 0x1a660ef 0x1a66 1ee 0x1a223a1 0x109741f 0x1093893 0xfbad38 0x9970dd 0x99acbd 0x393e21ed1d 0x9937e9 ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"F37CB2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F36BE9"},{"b":"400000","o ":"F373F2"},{"b":"393E600000","o":"F7E0"},{"b":"393E200000","o":"325E5","s":"gsignal"},{"b":"393E200000","o":"33DC5","s":"a bort"},{"b":"400000","o":"EBEA52","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"CAF8D3"},{"b":"400000","o":"169F62C" ,"s":"__wt_eventv"},{"b":"400000","o":"169FAED","s":"__wt_err"},{"b":"400000","o":"169FED4","s":"__wt_panic"},{"b":"400000" ,"o":"15D18FC","s":"__wt_block_extlist_read"},{"b":"400000","o":"15D1E73","s":"__wt_block_extlist_read_avail"},{"b":"400000 ","o":"15CEE15","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15D2EB9"},{"b":"400000","o":"15EF92C","s":"__wt_btree_ open"},{"b":"400000","o":"16257C0","s":"__wt_conn_btree_open"},{"b":"400000","o":"169E357","s":"__wt_session_get_btree"},{" b":"400000","o":"169E899","s":"__wt_session_get_btree"},{"b":"400000","o":"169E9BB","s":"__wt_session_get_btree_ckpt"},{"b" :"400000","o":"1633DB8","s":"__wt_curfile_open"},{"b":"400000","o":"169B515"},{"b":"400000","o":"16660EF","s":"__wt_metadat a_cursor_open"},{"b":"400000","o":"16661EE","s":"__wt_metadata_cursor"},{"b":"400000","o":"16223A1","s":"wiredtiger_open"}, {"b":"400000","o":"C9741F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C93893"},{"b":"400000", "o":"BBAD38","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5970DD","s":"_ZN5mo ngo13initAndListenEi"},{"b":"400000","o":"59ACBD","s":"main"},{"b":"393E200000","o":"1ED1D","s":"__libc_start_main"},{"b":" 400000","o":"5937E9"}],"processInfo":{ "mongodbVersion" : "3.2.7", "gitVersion" : "4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2 ", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-642.1.1.el6.x86_64", "version" : "#1 SMP Tu e May 31 21:57:07 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "D6BB40ACB6CF EAA43C3A651DDD8E547FCFE19166" }, { "b" : "7FFD6B8EE000", "elfType" : 3, "buildId" : "7F719AF91EE951B4FCB6647E7868F95F766A61 6B" }, { "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "7C5A504A21B221F299B1C45B9ED9C2340AEC6AEB" }, { "pa th" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "3E83FD82B1AE56360B270189F636423FB3016594" }, { "path" : "/l ib64/librt.so.1", "elfType" : 3, "buildId" : "86FE5BC1F46B8F8AA9A7A479FF991900DB93F720" }, { "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "045D39F19533291EBD72D0EE0247F9D49BE2521E" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "bui ldId" : "B60EF3FCE5D4D2D8BAD2585D5CAAA1167B35DBFD" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9A6E4B DFA184364D81F7DFD789474C3FB8F98A00" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "B3BD4C596D72FCBE460 7C86FEEC14F47B46D0DCC" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "F14F3F5B55FC325FDFA7FB6F22F73CA1C69773 79" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "DB04BDF55FE66BADD1081D833C17B4881E0EEDB9" }, { "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "9B852585C66329AA02EFB28497E652A40F538E78" }, { "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "EF3AACAFD6BF71BB861F194C1559153FB0B020E2" }, { "path" : "/lib64/libcom_ err.so.2", "elfType" : 3, "buildId" : "152E2C18A7A2145021A8A879A01A82EE134E3946" }, { "path" : "/lib64/libk5crypto.so.3", " elfType" : 3, "buildId" : "DDE6774979156442185836150FC0785170F8001F" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buil dId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "A 23DAFBCE170763BF1E836A8B26113F9CD20E0DA" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B 139A0B431D3F06D82014AFF3251" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "DDF6449707FD4C49DDE32A293EE E9AC218BFC460" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236 " } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1337cb2] mongod(+0xF36BE9) [0x1336be9] mongod(+0xF373F2) [0x13373f2] libpthread.so.0(+0xF7E0) [0x393e60f7e0] libc.so.6(gsignal+0x35) [0x393e2325e5] libc.so.6(abort+0x175) [0x393e233dc5] mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12bea52] mongod(+0xCAF8D3) [0x10af8d3] mongod(__wt_eventv+0x42C) [0x1a9f62c] mongod(__wt_err+0x8D) [0x1a9faed] mongod(__wt_panic+0x24) [0x1a9fed4] mongod(__wt_block_extlist_read+0x6C) [0x19d18fc] mongod(__wt_block_extlist_read_avail+0x33) [0x19d1e73] mongod(__wt_block_checkpoint_load+0x3C5) [0x19cee15] mongod(+0x15D2EB9) [0x19d2eb9] mongod(__wt_btree_open+0xC7C) [0x19ef92c] mongod(__wt_conn_btree_open+0x140) [0x1a257c0] mongod(__wt_session_get_btree+0xE7) [0x1a9e357] mongod(__wt_session_get_btree+0x629) [0x1a9e899] mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a9e9bb] mongod(__wt_curfile_open+0x218) [0x1a33db8] mongod(+0x169B515) [0x1a9b515] mongod(__wt_metadata_cursor_open+0x5F) [0x1a660ef] mongod(__wt_metadata_cursor+0x7E) [0x1a661ee] mongod(wiredtiger_open+0x1541) [0x1a223a1] mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x46F) [0x109741f] mongod(+0xC93893) [0x1093893] mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xfbad38] mongod(_ZN5mongo13initAndListenEi+0x3DD) [0x9970dd] mongod(main+0x15D) [0x99acbd] libc.so.6(__libc_start_main+0xFD) [0x393e21ed1d] mongod(+0x5937E9) [0x9937e9] ----- END BACKTRACE ----- Aborted ramon.fernandez commented on Thu, 22 Sep 2016 15:54:10 +0000: Paddyzhu, after further internal discussion we'd like to examine your database files to make sure you haven't found a new bug. Is it possible for you to upload the contents of /ick/data2 via this private, secure upload portal? Also, can you please upload the full log file for this node? I want to see if there are other clues before the failed restart that can help us determine what happened here. Thanks, Ramón. ramon.fernandez commented on Thu, 22 Sep 2016 10:44:06 +0000: Paddyzhu, I've run a repair attempt at the files you uploaded and attached them to this ticket. Please extract them in /ick/data2 and try to start mongod again, but note that there's no guarantee that this will work – I'd strongly recommend you use a replica set and keep good backups. I'd also recommend you upgrade to 3.2.9 (the latest 3.2 patch release at the time of this writing) as there have been many fixes to MongoDB 3.2 since 3.2.3. Thanks, Ramón.

Steps to Reproduce