Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
I am running some tests with TLS/SSL certificates. I like to use client certificates (for TLS/SSL and authentication) in a ReplicaSet / Sharded Cluster. It works fine when I run it on standalone MongoDB It works fine when I use a common CA for cluster and client, i.e. net: port: 27037 bindIpAll: true tls: mode: preferTLS allowConnectionsWithoutCertificates: true certificateKeyFile: c:\MongoDB\config\mongo.server.pem CAFile: c:\MongoDB\config\mongo-ca.cer clusterFile: c:\MongoDB\config\mongo.member.pem works fine. However, I like to have different CA for cluster and client certificates.
JIRAUSER1265262 commented on Fri, 10 Jun 2022 13:44:31 +0000: Hi Wernfried, Thanks for taking the time to come back and share the extra information on that. I'll go ahead and close this ticket. Regards, Christopher JIRAUSER1257089 commented on Wed, 8 Jun 2022 11:07:16 +0000: After creating dozens of different certificates combinations I got it. clusterCAFile is used to verify certificates provided by the client - "normal" client as well as "membership" client connections. In my opinion the name is a bit misleading, I was thinking it is related to clusterFile, i. e. clusterCAFile is used to verify certificates from "membership" clients CAFile is used to verify certificates from "normal" clients Obviously, my assumption was wrong (although I don't see any use-case for the way it is implemented) You can close this ticket. Kind Regards Wernfried JIRAUSER1257089 commented on Wed, 8 Jun 2022 06:47:31 +0000: Made a few more test. Obviously the client-certificate ( --tlsCertificateKeyFile=mongo.client.pem ) is verified by Cluster-CA ( clusterCAFile: mongo.member-ca.cer ) set EVAL=db.runCommand({ connectionStatus: 1 }).authInfo.authenticatedUsers c:\MongoDB\config>openssl verify -CAfile mongo-ca.cer -show_chain -verify_name ssl_client mongo.client.pem mongo.client.pem: OK Chain: depth=0: C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, OU = MongoDB, OU = Client, CN = admin (untrusted) depth=1: C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongo-ca mongosh --norc --quiet "mongodb://admin:password@%HOST%/admin?authSource=admin&replicaSet=repSet" --eval "%EVAL%" --tls --tlsCAFile=mongo-ca.cer --tlsCertificateKeyFile=mongo.client.pem MongoServerSelectionError: connection to 127.0.0.1:27037 closed c:\MongoDB\config>openssl verify -CAfile mongo.member-ca.cer -show_chain -verify_name ssl_client mongo.client-x.pem mongo.client-x.pem: OK Chain: depth=0: C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, OU = MongoDB, OU = Client, CN = admin (untrusted) depth=1: C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongo.member-ca mongosh --norc --quiet "mongodb://admin:password@%HOST%/admin?authSource=admin&replicaSet=repSet" --eval "%EVAL%" --tls --tlsCAFile=mongo-ca.cer --tlsCertificateKeyFile=mongo.client-x.pem [ { user: 'admin', db: 'admin' } ] -> Success I read carefully the documentation: "The .pem file that contains the root certificate chain from the Certificate Authority used to validate the certificate presented by a client establishing a connection. {{}} net.tls.clusterCAFile lets you use separate Certificate Authorities to verify the client to server and server to client portions of the TLS handshake." What does it mean? I don't understand. For "server to client portions of the TLS handshake" the CA is defined at client side ( mongosh --tls --tlsCAFile=... ), so what is then the difference between net.tls.clusterCAFile and net.tls.CAFile? I also tried the opposite net: tls: mode: preferTLS allowConnectionsWithoutCertificates: true CAFile: c:\MongoDB\config\mongo.member-ca.cer certificateKeyFile: c:\MongoDB\config\mongo.server.pem clusterFile: c:\MongoDB\config\mongo.member.pem clusterCAFile: c:\MongoDB\config\mongo-ca.cer But this does not work either, the replica set member cannot connect, the node remains in SECONDARY state.
This is the relevant part of the config file: net: port: 27037 bindIpAll: true tls: mode: preferTLS allowConnectionsWithoutCertificates: true certificateKeyFile: c:\MongoDB\config\mongo.server.pem CAFile: c:\MongoDB\config\mongo-ca.cer clusterFile: c:\MongoDB\config\mongo.member.pem clusterCAFile: c:\MongoDB\config\mongo.member-ca.cer security: clusterAuthMode: x509 authorization: enabled Relevant information of the certificates: openssl x509 -in c:\MongoDB\config\mongo-ca.cer -noout -subject subject=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongo-ca openssl x509 -in c:\MongoDB\config\mongo.member-ca.cer -noout -subject subject=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongo.member-ca openssl x509 -in c:\MongoDB\config\mongo.member.pem -noout -issuer -subject -ext extendedKeyUsage,keyUsage issuer=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongo.member-ca subject=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = mongodb.member, CN = mongodb.member X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Client Authentication openssl x509 -in c:\MongoDB\config\mongo.server.pem -noout -issuer -subject -ext extendedKeyUsage,keyUsage issuer=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongo-ca subject=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = mongodb.server, CN = mongodb.server X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication Certificates are valid: openssl verify -CAfile mongo.member-ca.cer -verify_name ssl_client mongo.member.pem mongo.member.pem: OK openssl verify -CAfile mongo-ca.cer -verify_name ssl_server mongo.server.pem mongo.server.pem: OK Start Replica Set members. When the second member starts then I get this error on the first one: { "t": { "$date": "2022-06-03T15:32:15.267+02:00" }, "s": "I", "c": "ACCESS", "id": 20428, "ctx": "conn898", "msg": "Failed to authenticate", "attr": { "client": "10.80.41.19:64138", "mechanism": "MONGODB-X509", "user": "CN=mongodb.member,OU=mongodb.member,O=Sunrise-UPC,L=Zurich,ST=ZH,C=CH", "db": "$external", "error": { "code": 11, "codeName": "UserNotFound", "errmsg": "Could not find user \"CN=mongodb.member,OU=mongodb.member,O=Sunrise-UPC,L=Zurich,ST=ZH,C=CH\" for db \"$external\"" } } } and member remains in SECONDARY state. Apparently the replica set considers the member connection as a normal client connection, which is not the case. (I am not try to connect with any client) I tied with different certificates (matching O and OU): openssl x509 -in c:\MongoDB\config\mongo.member.pem -noout -issuer -subject -ext extendedKeyUsage,keyUsage issuer=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongo.member-ca subject=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongodb.member X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Client Authentication openssl x509 -in c:\MongoDB\config\mongo.server.pem -noout -issuer -subject -ext extendedKeyUsage,keyUsage issuer=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongo-ca subject=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongodb.server X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication With these certificates the ReplicaSet starts up as expected: { "t": { "$date": "2022-06-03T15:57:21.081+02:00" }, "s": "I", "c": "ACCESS", "id": 20429, "ctx": "conn3", "msg": "Successfully authenticated", "attr": { "client": "10.80.41.19:51039", "mechanism": "MONGODB-X509", "user": "CN=mongodb.member,OU=OSS,O=Sunrise-UPC,L=Zurich,ST=ZH,C=CH", "db": "$external" } } However, I get this warning which actually no problem - but it is not true. The client is another mongod repliaset member, thus this warning should not appear: { "t": {"$date": "2022-06-03T15:57:21.081+02:00" }, "s": "W", "c": "ACCESS", "id": 20430, "ctx": "conn3", "msg": "Client isn't a mongod or mongos, but is connecting with a certificate with cluster membership" } Anyway, I tried several client certificates, none of them is working: openssl x509 -in c:\MongoDB\config\mongo.client.pem -noout -issuer -subject -ext extendedKeyUsage,keyUsage issuer=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = OSS, CN = mongo-ca subject=C = CH, ST = ZH, L = Zurich, O = Sunrise-UPC, OU = ClientAuthentication, CN = admin X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Client Authentication openssl x509 -in mongo.client.pem -noout -subject -nameopt RFC2253 subject=CN=admin,OU=ClientAuthentication,O=Sunrise-UPC,L=Zurich,ST=ZH,C=CH db.getSiblingDB("$external").runCommand({ createUser: "CN=admin,OU=ClientAuthentication,O=Sunrise-UPC,L=Zurich,ST=ZH,C=CH", roles: [{ role: "root", db: "admin" }] }) I cannot use the client certificate, neither for TLS/SSL nor for authentication: mongosh --norc --quiet "mongodb://localhost:27037/admin?authSource=$external" --tls --tlsCertificateKeyFile=mongo.client.pem --tlsCAFile=mongo-ca.cer --authenticationMechanism=MONGODB-X509 MongoServerSelectionError: connection to 127.0.0.1:27037 closed mongosh --norc --quiet "mongodb://admin:password@localhost:27037/admin?authSource=admin" --tls --tlsCertificateKeyFile=mongo.client.pem --tlsCAFile=mongo-ca.cer MongoServerSelectionError: connection to 127.0.0.1:27037 closed Note, the same works fine on a Standalone MongoDB, it only fails on a cluster/replica set: mongosh --norc --quiet "mongodb://localhost:27017/admin?authSource=$external" --eval "db.getMongo()" --tls --tlsCertificateKeyFile=mongo.client.pem --tlsCAFile=mongo-ca.cer --authenticationMechanism=MONGODB-X509 mongodb://localhost:27017/admin?authSource=%24external&directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+1.3.1 mongosh --norc --quiet "mongodb://admin:password@localhost:27017/admin?authSource=admin" --eval "db.getMongo()" --tls --tlsCertificateKeyFile=mongo.client.pem --tlsCAFile=mongo-ca.cer mongodb://@localhost:27017/admin?authSource=admin&directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+1.3.1 It also works fine if I use a common CA for cluster and client, i.e. net: port: 27037 bindIpAll: true tls: mode: preferTLS allowConnectionsWithoutCertificates: true certificateKeyFile: c:\MongoDB\config\mongo.server.pem CAFile: c:\MongoDB\config\mongo-ca.cer clusterFile: c:\MongoDB\config\mongo.member.pem works fine. However, I like to have different CA for cluster and client certificates. Is this a bug or do I anything wrong?