MongoDB - Defect ID: 2915104
Investigate filling up disk space with core dumps

Info

Original slack thread: https://mongodb.slack.com/archives/C07T35KELLC/p1738125749813369 Max: color: Color value is invalid looks like the hang analyzer taking core dumps is filling up 100GB in https://jira.mongodb.org/browse/BF-36363 color: Color value is invalid . From the system logs in  https://parsley.mongodb.com/evergreen/mongodb_mongo_master_enterprise_amazon_linux2_ar[…]_16_41_34/0/system?bookmarks=0,15107&filters=100mongod%2520 color: Color value is invalid , I see the total virtual memory used by all 24 mongod processes is 99GB. The resident memory is less than 5GB. I feel like one avenue would be to investigate what allocated memory within the mongod process because the WT cache is still restricted to 1GB. This is maybe better performed by the Server team so I'll tag  color: Color value is invalid @Abdul Qadeer color: Color value is invalid  as well color: Color value is invalid For the Correctness team, I think  https://github.com/10gen/mongo/blob/6614f6533ef5932a95dd1b9b2eda8ff280a7a548/buildscripts/resmokelib/hang_analyzer/hang_analyzer.py#L174 color: Color value is invalid  needs to be looked into for why it is ineffective at preventing the disk from filling up with core dumpsFor the future, I think the Correctness team can also consider using a named pipe so the contents of the  gcore color: Color value is invalid  command can be piped into gzip/pgzip since the compressed size of the core dumps will be much smaller

Top User Comments

Steps to Reproduce