Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
After replacing certificates on vCenter Server, you experience these symptoms: ESX Agent Manager solution user fails to log in.In the /var/log/vmware/eam/eam.log (on VCSA) file or the C:\ProgramData\VMware\vCenterServer\logs\eam\eam.log (on Windows) file for the ESX Agent Manager (EAM), you see entries similar to: YYYY-MM-DDTHH:MM:SS.MSZ | INFO | eam-0 | VcConnection.java | 167 | Connecting to vCenter as com.vmware.vim.eam extension YYYY-MM-DDTHH:MM:SS.MSZ | INFO | eam-0 | VcConnection.java | 603 | Connecting to https://vCenter_Server_FQDN:8089/sdk/vimService via vCenter proxy http://localhost:80YYYY-MM-DDTHH:MM:SS.MSZ | DEBUG | http-bio-0.0.0.0-15005-exec-1 | AllowAllSamlTokenPolicy.java | 24 | HealtStatus request's token subject name: machine-7502fb4c-3521-48c7-93ed-3d1865e0fff1, subject domain: vsphere.localYYYY-MM-DDTHH:MM:SS.MSZ | ERROR | eam-0 | VcConnection.java | 179 | Failed to login to vCenter as extension. vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password.YYYY-MM-DDTHH:MM:SS.MSZ | WARN | eam-0 | VcListener.java | 114 | Trying to recover from error (vim.fault.InvalidLogin) { faultCause = null, faultMessage = null } at sun.reflect.GeneratedConstructorAccessor82.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at java.lang.Class.newInstance(Unknown Source) at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:173) at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:26) at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:31) at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:141) at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:102) at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:89) at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:84) at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:41) at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:112) at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:273) at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230) at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:144) at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51) at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:186) at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:77) at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:581) at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:562) at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:348) at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:308) at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:182) at com.sun.proxy.$Proxy48.loginExtensionByCertificate(Unknown Source) at com.vmware.eam.vc.VcConnection.connectEam(VcConnection.java:171) at com.vmware.eam.vc.VcListener.login(VcListener.java:149) at com.vmware.eam.vc.VcListener.main(VcListener.java:129) at com.vmware.eam.vc.VcListener.call(VcListener.java:111) at com.vmware.eam.vc.VcListener.call(VcListener.java:60) at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:35) at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) YYYY-MM-DDTHH:MM:SS.MSZ | INFO | eam-0 | VcListener.java | 121 | Retrying in 10 Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment. Unable to deploy VIBs to your ESXi hosts from NSX for vSphere or vCloud Networking and Security.vCenter Server experiences high CPU usage.
This issue can happen if there is a mismatch between vpxd-extension certificate stored in VECS and the certificate information stored in vCenter Server Database for EAM extension.
To resolve the issue, update the extension's certificate with vCenter Server by following below steps. Notes: Certificate Manager automatically updates the certificate for extensions like com.vmware.vim.eam (EAM), com.vmware.rbd (Autodeploy) from vCenter Server 6.0 U1b onwards. However, below steps can be tried on any vCenter Server 6.x or 7.x versions if you are seeing the error messages mentioned in Symptoms sectionPlease refer to the Related Information in this KB if you are receiving the error "certificate verify failed: Hostname mismatch, certificate is not valid for 'sdkTunnel'" during Extension certificate registration Process to update the extension's certificate: On vCenter Server for WindowsOn the vCenter Server Appliance On vCenter Server for Windows: Connect to vCenter Server through a Console or Remote desktop session.Open an elevated command prompt.Create a temporary directory named c:\certificate.Run this command to retrieve the vpxd-extension solution user certificate and key: "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output c:\certificate\vpxd-extension.crt "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output c:\certificate\vpxd-extension.key Navigate to C:\Program Files\VMware\vCenter Server\vpxd\scripts: cd C:\Program Files\VMware\vCenter Server\vpxd\scripts Note: The path listed is for a default install of vCenter Server. If you have customized the install location of vCenter Server, change the directory accordingly. Run this command to update the extension's certificate with vCenter Server: "%VMWARE_PYTHON_BIN%" updateExtensionCertInVC.py -e com.vmware.vim.eam -c C:\Certificate\vpxd-extension.crt -k C:\Certificate\vpxd-extension.key -s <PNID/FQDN of vCenter Server> -u Administrator@vsphere.local Note: The default user and domain is Administrator@vsphere.local. If this was changed during configuration, change the domain to match your environment. When prompted, enter the Administrator@vsphere.local password.Navigate to the C:\Program Files\VMware\vCenter Server\bin folder: cd C:\Program Files\VMware\vCenter Server\bin Restart the VMware ESX Manager service with these commands: For more information on managing service in the vCenter Server Appliance, see Stopping, starting, or restarting VMware vCenter Server Appliance 6.0 services (2109887). service-control --stop EsxAgentManager service-control --start EsxAgentManager On the vCenter Server Appliance: Log in to the vCenter Server Appliance using SSH.Run this command to enable access the Bash shell (Note: this is applicable only on vCenter Server 6.0): shell.set --enabled true Type shell and press Enter.Run this command to retrieve the vpxd-extension solution user certificate and key: mkdir /certificate /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key Run this command to update the extension's certificate with vCenter Server. python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <PNID/FQDN of vCenter Server> -u Administrator@vsphere.local Note: The default user and domain is Administrator@vsphere.local. If this was changed during configuration, change the domain to match your environment. When prompted, type in the Administrator@vsphere.local password. Restart the VMware ESX Manager service with these commands: service-control --stop vmware-eam service-control --start vmware-eam For more information on managing service in the vCenter Server, see Stopping, starting, or restarting VMware vCenter Server 6.0 services (2109881).
In certain situations, you might receive the error "certificate verify failed: Hostname mismatch, certificate is not valid for 'sdkTunnel'". This error can be safely ignored if you are getting the error after the message "Successfully updated certificate for "com.vmware.vim.eam" extension" as this message confirms that Extension certificate updated successfully with vCenter Server. root@hostname [ ~ ]# python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s vcsa1.test.com -u Administrator@vsphere.localPassword to connect to VC server for user="Administrator@vsphere.local":2021-03-11T22:31:22.517Z Updating certificate for "com.vmware.vim.eam" extension2021-03-11T22:31:22.649Z Successfully updated certificate for "com.vmware.vim.eam" extensionTraceback (most recent call last): File "/usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py", line 175, in <module> update_extension_cert_in_VC() File "/usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py", line 163, in update_extension_cert_in_VC sessionMgr = si.content.sessionManager File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 577, in __call__ return self.f(*args, **kwargs) File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 382, in _InvokeAccessor return self._stub.InvokeAccessor(self, info) File "/usr/lib/vmware/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor return self.InvokeMethod(mo, info, (prop, )) File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1525, in InvokeMethod conn.request('POST', self.path, req, headers) File "/usr/lib/python3.7/http/client.py", line 1277, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.7/http/client.py", line 1323, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.7/http/client.py", line 1272, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.7/http/client.py", line 1032, in _send_output self.send(msg) File "/usr/lib/python3.7/http/client.py", line 972, in send self.connect() File "/usr/lib/python3.7/http/client.py", line 1447, in connect server_hostname=server_hostname) File "/usr/lib/python3.7/ssl.py", line 423, in wrap_socket session=session File "/usr/lib/python3.7/ssl.py", line 870, in _create self.do_handshake() File "/usr/lib/python3.7/ssl.py", line 1139, in do_handshake self._sslobj.do_handshake()ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'sdkTunnel'. (_ssl.c:1076) VMware Skyline Health Diagnostics for vSphere - FAQvecs-cli failed. Error 87: Operation failed with error ERROR_INVALID_PARAMETER (87)How to stop, start, or restart vCenter Server 6.x servicesResolution: Internal error occurs during vSphere ESX Agent Manager pre-upgrade checks"WARNING: VMware ESX Agent Manager may have failed to start", EAM Service fails to start after vCenter Server rebootRead this article in different language here:VMware vSphere 6.0 で vCenter Server 証明書を置き換えると ESX Agent Manager ソリューション ユーザーがログインに失敗するDepois de substituir os certificados do vCenter Server no VMware vSphere 6.0, o usuário de solução ESX Agent Manager não consegue fazer loginDespués de reemplazar los certificados de vCenter Server en VMware vSphere 6.0, el usuario de la solución de ESX Agent Manager no puede iniciar sesión在 VMware vSphere 6.0 中替换 vCenter Server 证书后，ESX Agent Manager 解决方案用户无法登录