Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist

MongoDB - Defect ID: 2007924

Unable to remove unreachable members from the replica set.

Last updated on 5/16/2022

Overall
6.26.2
Severity
6.46.4
Community
66.0
Lifecycle
9.19.1

Vendor details

  • Priority: Major - P3
  • Status: Closed

Info

A replica set deployed in an kubernetes environment, and tried to add two nodes which are unreachable were and added, but rs.remove() does not remove it once when logged in inside pod using mongo. conf = { _id: "replset", members: [ {_id: 0, host: "localhost:27017"} ] } rs.initiate(conf)// Adding and removing one nonexistent secondary should work assert.commandWorked(rs.add("localhost:28018")) assert.commandWorked(rs.remove("localhost:28018"))// Adding and removing two nonexistent secondaries does not work assert.commandWorked(rs.add("localhost:28018")) assert.commandWorked(rs.add("localhost:29019")) assert.commandWorked(rs.remove("localhost:28018")) assert.commandWorked(rs.remove("localhost:29019")) Note: This does not depend on the k8 environment and this issue doesn't seem to affect mongosh but only the mongo shell. I tested this on MongoDB v5.0.7.

Top User Comments

eric.sedor commented on Mon, 16 May 2022 20:20:09 +0000: Thanks aasawari.sahasrabuddhe@mongodb.com. Since the mongo shell is deprecated and this works in mongosh, I'm going to close out this ticket. We really recommend that users move to mongosh wherever possible. JIRAUSER1264574 commented on Mon, 16 May 2022 07:26:55 +0000: Hi Eric, Apologies for the delay. I have uploaded the requested files. The nodes are unreachable because I created a new replica set and added two nodes that do not exist. Please note that this seems to affect only the mongo shell and does not seem to affect mongosh. eric.sedor commented on Wed, 11 May 2022 18:48:58 +0000: Hi aasawari.sahasrabuddhe@mongodb.com, I've created a secure upload portal for you. Files uploaded to this portal are hosted on Box, are visible only to MongoDB employees, and are routinely deleted after some time. Can you perform a reproduction of this and then, for each node in the replica set, would you please archive (tar or zip) and upload to that link: the mongod logs the $dbpath/diagnostic.data directory (the contents are described here) Can you also clarify how the nodes have been made to be "unreachable"? JIRAUSER1264574 commented on Thu, 24 Mar 2022 10:21:41 +0000: The rs.remove() does work when logged in to the pod using mongosh kubectl exec -it -n namespace mongosh  but doesnt seem to work in mongo.  Secondly, in mongo mode, the splice and rs.reconfig() has worked for my case.

Steps to Reproduce

Login to any MongoDB pod in a kubernetes environment:  kubectl exec it - namespace mongo Create a replica set with a primary member. add an unreachable node rs.add("mongo-1-mongo:27017") rs.add("mongo-2-mongo:27017") { { "_id" : 7, "name" : "mongo-1-mongo:27017", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "ts" : Timestamp(0, 0), "t" : NumberLong(-1) }, "optimeDurable" : { "ts" : Timestamp(0, 0), "t" : NumberLong(-1) }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"), "lastAppliedWallTime" : ISODate("1970-01-01T00:00:00Z"), "lastDurableWallTime" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2022-03-24T10:04:04.439Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : NumberLong(0), "lastHeartbeatMessage" : "Error connecting to mongo-1-mongo:27017 :: caused by :: Could not find address for mongo-1-mongo:27017: SocketException: Host not found (authoritative)", "syncSourceHost" : "", "syncSourceId" : -1, "infoMessage" : "", "configVersion" : -1, "configTerm" : -1 }, { "_id" : 8, "name" : "mongo-2-mongo:27017", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "ts" : Timestamp(0, 0), "t" : NumberLong(-1) }, "optimeDurable" : { "ts" : Timestamp(0, 0), "t" : NumberLong(-1) }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"), "lastAppliedWallTime" : ISODate("1970-01-01T00:00:00Z"), "lastDurableWallTime" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2022-03-24T10:04:04.433Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : NumberLong(0), "lastHeartbeatMessage" : "Error connecting to mongo-2-mongo:27017 :: caused by :: Could not find address for mongo-2-mongo:27017: SocketException: Host not found (authoritative)", "syncSourceHost" : "", "syncSourceId" : -1, "infoMessage" : "", "configVersion" : -1, "configTerm" : -1 } ], The two nodes should be unreachable nodes and then do rs.remove("mongo-1-mongo:27017") rs.remove("mongo-2-mongo:27017") rs0:PRIMARY> rs.remove("mongo-2-mongo:27017"){ "ok" : 0, "errmsg" : "Cannot provide newlyAdded field to member config during reconfig.", "code" : 93, "codeName" : "InvalidReplicaSetConfig", "$clusterTime" : { "clusterTime" : Timestamp(1648116296, 1), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } }, "operationTime" : Timestamp(1648116296, 1)} rs0:PRIMARY> rs.remove("mongo-1-mongo:27017"){ "ok" : 0, "errmsg" : "Cannot provide newlyAdded field to member config during reconfig.", "code" : 93, "codeName" : "InvalidReplicaSetConfig", "$clusterTime" : { "clusterTime" : Timestamp(1648116306, 1), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } }, "operationTime" : Timestamp(1648116306, 1)} rs0:PRIMARY>

  • 5.9Defect ID: 2956672
    Some time-series tests implicitly rely on measurement insertion order for unordered inserts when checking bucket catalog stats
  • 6.14Defect ID: 2965528
    Remove push, publish_packages, and crypt_push tasks from Graviton 4 variants in v7.0 and v8.0
  • 6.14Defect ID: 2947969
    [SBE] Release storage engine resources when saveState() or restoreState() throws
  • 5.68Defect ID: 2919474
    StackLocator broken by v5 toolchain ASAN
  • 5.88Defect ID: 2968769
    Make new write path helper functions use acquireAndValidateBucketsCollection instead of acquireCollection

Ready to prevent the next vendor outage?

BugZero | MongoDB BugID 2007924 - Unable to remove unreachable members from the repl...