OPERATIONAL DEFECT DATABASE
...
...
1. Connectivity to the vCenter is lost.2. Rediscover of the vCenter fails, also we cannot remove the vCenter.3. Resetting the credentials pops dialog box with below error: Exception in VCConnectionCache:failed to connect: Keystore was tampered with, or password was incorrect : Password verification failed Reviewing the Server logs with respect to timestamp of trial shows events as below: DEBUG [default task-29] [com.emc.archway.securedcommands.ArchwayEJBInterceptor] [Appsync Server Name] [] 0dbfcdd0-69a8-43b2-9555-e8133aa932ca->>> Bean level security checking for public java.util.UUID com.emc.archway.securedcommands.vmware.ValidateCreateVCenterServerCommandSecuredBean.validateAndCreate(com.emc.archway.context.ExecutionContext,com.emc.archway.objectmodel.host.vcenterserver.VCenterServer) throws java.rmi.RemoteException,com.emc.archway.errorMessages.ObjectAlreadyExistsException,com.emc.archway.errorMessages.ArchwayException,com.emc.archway.acl.AuthorizationConfigException DEBUG [default task-29] [com.emc.archway.service.vc.VCEntityLockCommandBean] [Appsync Server Name] [] 0dbfcdd0-69a8-43b2-9555-e8133aa932ca->>> EJB default - 1Locking entityID:https://FQDN/sdk for update INFO [default task-29] [com.emc.archway.service.vc.VCConnectionCache] [Appsync Server Name] [] 0dbfcdd0-69a8-43b2-9555-e8133aa932ca->>> getURLConnection::https://FQDN/sdk creating a new connection to url:https://vmcsxxxx.xxx.co.xx/sdk as no cached connection available INFO [default task-29] [com.emc.archway.service.vc.BasicVCConnection] [Appsync Server Name] [] 2b3e6a5c-f7e2-41f7-ba01-229fcb6d6417->>> establishing TLS connection DEBUG [default task-29] [com.emc.archway.service.vc.EnableSecurity] [Appsync Server Name] [] 2b3e6a5c-f7e2-41f7-ba01-229fcb6d6417->>> getTrustMangerForCertificates using keystoreName: install_dir:\EMC\AppSync\jboss\_jre\lib\security\cacerts ERROR [default task-29] [com.emc.archway.service.vc.VCConnectionCache] [Appsync Server Name] [] 0dbfcdd0-69a8-43b2-9555-e8133aa932ca->>> Exception in VCConnectionCache:failed to connect: Keystore was tampered with, or password was incorrect : Password verification failed INFO [default task-29] [com.emc.archway.service.eventservice.EventServiceBean] [Appsync Server Name] [] 0dbfcdd0-69a8-43b2-9555-e8133aa932ca->>> EVENT [VM_000156]: Unable to create a new connection to vCenter server:https://FQDN/sdk failed to connect: Keystore was tampered with, or password was incorrect : Password verification failed(METADATA: TYPE-ERROR, TIME-Time-Stamp.738+0400NATIVETIME-2022-05-17 , HOST-Appsync Server Name, PHASE-, THREAD=default task-29, USER-s911987, CATEGORY-GENERIC, SESSIONID-iGywYTLkg9bspkRYZYlvF9gD8vP78L0__2c6_0N2) INFO [default task-29] [com.emc.archway.service.vc.VCEntityLockCommandBean] [Appsync Server Name] [] 0dbfcdd0-69a8-43b2-9555-e8133aa932ca->>> EJB default - 1Unlocking entityID:https://FQDN/sdk for update INFO [default task-29] [com.emc.archway.service.vc.VCServiceBean] [Appsync Server Name] [] 0dbfcdd0-69a8-43b2-9555-e8133aa932ca->>> Exception while getting connection to vc ::Unable to create a new connection to vCenter server:https://FQDN/sdk. Error: failed to connect: Keystore was tampered with, or password was incorrect : Password verification failed: com.emc.archway.service.eventservice.exceptions.ConnectionManagerException: Unable to create a new connection to vCenter server:https://FQDN/sdk. Error: failed to connect: Keystore was tampered with, or password was incorrect : Password verification failed at deployment.archway-ear.ear.vcservice-4.3.0.0-SNAPSHOT.jar//com.emc.archway.service.vc.VCConnectionCache.getConnection(VCConnectionCache.java:236)
In the above case, a third party application happened to corrupt the cacert file leading to the cause of the issue.
Follow the below steps to validate and replace the corrupted cacer file with the healthy cacert file.1. To Validate if the ca-cert file is corrupted. Run the following command in the cmd(admin) on the AppSync server: install_dir:\EMC\AppSync\jre\bin>keytool.exe -list -v -keystore Install_dir:\emc\appsync\jboss\_jre\lib\security\cacerts (enter changeit for the pwd). If the above command errors out with the "Keystore was tampered with, or password was incorrect : Password verification failed:" error, then the file is corrupted and needs to be replaced, hence Continue. 2. To validate if the replacing file is healthy. Run the following command in the cmd(admin) on the AppSync server: install_dir:\EMC\AppSync\jre\bin>keytool.exe -list -v -keystore install_dir:\emc\appsync\jre\lib\security\cacerts (enter changeit for the pwd) The above command should execute successfully. (If this errors out, need to contact Support). 3. Replace the corrupted file with a healthy one: Run the following command in the cmd(admin) on the AppSync server copy install_dir:\EMC\AppSync\jre\lib\security\cacerts install_dir:\EMC\AppSync\jboss\_jre\lib\security if above command does not work copy manually cacerts from: install_dir:\EMC\AppSync\jre\lib\security\cacerts to install_dir:\EMC\AppSync\jboss\_jre\lib\security 4. Import the newly replaced ca-cert file. Run the following command in the cmd(admin) on the AppSync server install_dir:\EMC\AppSync\jboss\_jre\bin>keytool.exe -importcert -file install_dir:\EMC\AppSync\jboss\_jre\lib\security\cas.crt -keystore install_dir:\EMC\AppSync\jboss\_jre\lib\security\cacerts -storepass changeit -alias appsync -noprompt 5. Restart the services and try to login to AppSync Note: Need to replace the install_dir with the AppSync installation directory wherever mentioned.(Recommended to take the back up of cacerts from install_dir:\EMC\AppSync\jboss\_jre\lib\security so that we can avoid running the same commands again if the file is corrupted again.)
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
