Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
PSC Replication fails due to invalid credential of machine account in VMware Directory Service (vmdird) database/var/log/vmware/vmdird/vmdird-syslog.log contains the following entries: 18-02-28T13:52:12.965331-05:00 err vmdird t@140245530842880: Bind Request Failed (x.x.x.x) error 49: Protocol version: 3, Bind DN: "cn=vcsa1,ou=Domain Controllers,dc=domain,dc=local", Method: SASL 18-02-28T13:52:13.078809-05:00 err vmdird t@140245530842880: SASLSessionStep: sasl error (-13)(SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error) Inventory Service in vCenter Server 6.0 / VPXD-SVCS service in vCenter Server 6.5/6.7 or 7.0 fails to startthe service log (/var/log/vmware/invsvc/inv-svc.log in vCenter Server 6.0 or /var/log/vmware/vpxd-svcs/vpxd-svcs.log in vCenter Server 6.5/6.7 or 7.0 contains the following message: Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.cis.core.authz.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials LDAP error [code: 49] Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
This KB will help to reset machine account password in VMware Directory Services Database using a shell script, it is applicable for vCenter Server Appliance only.
This issue happens due to a mismatch in the machine account password stored in VMDIRD Database and the password used by services to connect to VMware Directory Service.
The article also assumes you have taken powered-off snapshots of all the vCenter Server or PSC nodes in the same vSphere Domain (ELM) prior to attempting the fix (per the instructions set forth in the resolution section of this article).Should something go wrong, you will have to restore the snapshots taken before the attempted fix.
NOTE: Please take offline (powered off) snapshots of all PSC's and VC's in the same vSphere Domain (or in ELM) before attempting. This is standard best practice before making any manual changes to the PSC VMDIRD database. Copy the script attached to this article on the vCenter Server or PSC which is facing the issue with Invalid Credentialsverify that the vmdir database is in normal state: # /usr/lib/vmware-vmafd/bin/dir-cli state get The output should look like this: Directory Server State: Normal (3) If the vmdir database is not in normal state, change it by running: # /usr/lib/vmware-vmafd/bin/dir-cli state set --state NORMAL Make the script executable by executing the following command: # chmod +x reset_machine_pw.sh Run the script. You will be prompted for the Administrator@<sso.domain> password and replication partner name if it is executed on PSC or Embedded NodeRestart all the services after modifying the password: # service-control --stop --all && service-control --start --all Note: This script needs to be executed on VMDIR replication partner node as well, if replication is not working in both directions due to invalid credentials of each other's machine account. Example output : Script executed on a Platform Services Controller with a replication partner: a) Identify the partner nodes using vdcrepadmin command: root@vcsa1 [ /tmp ]# /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator password: Partner: vcsa2.domain.local b) Execute the script to reset password root@vcsa1 [ /tmp ]# ./reset_machine_pw.sh ================================== Machine account password reset for vcsa1.domain.local started on Wed Jun 19 09:09:49 UTC 2019 Detected that this node is an external PSC. Please provide the replication partners separated by a space: vcsa2.domain.local Detected DN: cn=vcsa1.domain.local,ou=Domain Controllers,dc=vsphere,dc=local Detected PNID: vcsa1.domain.local Detected PSC: vcsa1.domain.local Detected SSO domain name: vsphere.local Enter password for administrator@vsphere.local: updating registry with password. updating local PSC with password. modifying entry "cn=vcsa1.domain.local,ou=Domain Controllers,dc=vsphere,dc=local" Updating replication partners with the new password as well. Changing password for vcsa1.domain.local in the VMDIR database located at vcsa2.domain.local modifying entry "cn=vcsa1.domain.local,ou=Domain Controllers,dc=vsphere,dc=local" Finished on Wed Jun 19 09:09:57 UTC 2019 c) Execute the script on Partner Node as well if VMDIR replication is not working in both directions due to error 49. In above example, same script needs to be executed on partner node vcsa2.domain.local Script executed on a vCenter Server Node with an external PSC a) Execute the script to reset the password root@vcsaext[ /tmp ]# ./reset_machine_pw.sh ================================== Machine account password reset for vcsaext.domain.local started on Wed Jun 19 09:19:32 UTC 2019 Detected this node is a vCenter server with external PSC. Detected DN: cn=vcsaext.domain.local,ou=Computers,dc=vsphere,dc=local Detected PNID: vcsaext.domain.local Detected PSC: psc.domain.local Detected SSO domain name: vsphere.local Enter password for administrator@vsphere.local: updating registry with password. updating local PSC with password. modifying entry "cn=vcsaext.domain.local,ou=Computers,dc=vsphere,dc=local" Since there were no replication partners specified, we're done here. Finished on Wed Jun 19 09:19:38 UTC 2019
NOTE: You may receive an error when you try to run the script: bash: ./reset_machine_pw.sh: /bin/bash^M: bad interpreter: No such file or directory This error is caused by DOS carriage returns added to the script when copying from a Windows-based text editor. To resolve this problem: run the following command: # sed -i -e 's/\r$//' reset_machine_pw.sh Rerun the script.