Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
Attempting to the vCenter certificate replacement workflow fails with below error in the /var/log/vmware/vcf/operationsmanager/operationsmanager.log: 2021-05-11T05:07:00.287+0000 ERROR [vcf_om,6c92de84a35548da,391e] [c.v.v.c.vc.VCenterCertificatePlugin,om-exec-29] Replace certificate failed for resource: {"version":"7.0.2.00000-17694817","hostName":"<FQDN>","id":"54df8203-2e72-40b4-ba51-d1dc4af443e3","vmName":"<VM_NAME>","resourceType":"vcenter","credentials":[{"username":"root","secret":"*****","type":"ssh"},{"username":"administrator@vsphere.local","secret":"*****","type":"sso"}],"ipAddress":"<IP>","sans":["<FQDN>"],"master":false} : 500 Internal Server Error: [{"type":"com.vmware.vapi.std.errors.error","value":{"error_type":"ERROR","messages":[{"args":["AFD Native Error Occured: 11"],"default_message":"Exception found (AFD Native Error Occured: 11)","id":"com.vmware.certificatemanagement.error"}]}}] : {}com.vmware.vcf.certmgmt.exceptions.CertificateReplacementException: 500 Internal Server Error: [{"type":"com.vmware.vapi.std.errors.error","value":{"error_type":"ERROR","messages":[{"args":["AFD Native Error Occured: 11"],"default_message":"Exception found (AFD Native Error Occured: 11)","id":"com.vmware.certificatemanagement.error"}]}}] at com.vmware.vcf.certmgmt.vc.VCenterCertificatePluginService.replaceCertificate(VCenterCertificatePluginService.java:230) at com.vmware.vcf.certmgmt.vc.VCenterCertificatePluginService.uploadCertificate(VCenterCertificatePluginService.java:179) at com.vmware.vcf.certmgmt.vc.VCenterCertificatePlugin.replaceCertificate(VCenterCertificatePlugin.java:83)at com.vmware.vcf.certmgmt.vc.VCenterCertificatePlugin$$FastClassBySpringCGLIB$$874f9739.invoke(<generated>)at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) The SDDC Manager and vCenter re-trust sub task fails with below error: 2021-05-11T05:15:22.959+0000 ERROR [vcf_om,6c92de84a35548da,701f] [c.v.v.c.v.VCenterCertificatePluginService,om-exec-25] 500 Internal Server Error: [{"type":"com.vmware.vapi.std.errors.internal_server_error","value":{"error_type":"INTERNAL_SERVER_ERROR","messages":[{"args":["com.vmware.vapi.std.errors.Error"],"default_message":"Provider method imp... (482 bytes)]org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 Internal Server Error: [{"type":"com.vmware.vapi.std.errors.internal_server_error","value":{"error_type":"INTERNAL_SERVER_ERROR","messages":[{"args":["com.vmware.vapi.std.errors.Error"],"default_message":"Provider method imp... (482 bytes)] at org.springframework.web.client.HttpServerErrorException.create(HttpServerErrorException.java:100) at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:186) at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:125) at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:780) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:738) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:712) at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:600) Running /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh on the vCenter reports the following Error: Failed to trigger root cert refreshvecs-cli failed. Error 11: Possible errors:LDAP error: Administrative limit exceededWin Error: Operation failed with error ERROR_BAD_FORMAT (11) Adding a trusted root certificate to Certificate Management via UI fails with "Error occurred while adding trusted root certificates" Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.
If the vCenter certificate replacement fails or the SDDC Manager and vCenter re-trust fails the vCenter is unable to process any certificate related workflows, which can lead other workflows like add/ remove host, etc to fail.
This issue occurs if root certificate content is of length in the order of 233 multiples.
This is a known issue affecting all VCF 4.x releases and will be resolved in a future release.
To workaround this problem, please either use the attached fixCertRetrust.sh script or follow the steps below in Option 2.Option 1: Download the fixCertRetrust.sh script.Using a file transfer utility, like WinSCP, copy the file to /tmp/ on the the vCenter appliance.Open an SSH session to the vCenter appliance/Change the permssions on the file: chmod +x fixCertRetrust.sh Run the script: sh CertRepublish.sh Option 2: Open an SSH session to the vCenter appliance with the root user. Get the trusted root certificates list /usr/lib/vmware-vmafd/bin/dir-cli trustedcert list --login administrator@<domain> Take a backup of all the certificates; using the IDs the output from step 1 /usr/lib/vmware-vmafd/bin/dir-cli trustedcert get --id <ID> --outcert trustedCert_x.crt If there is no new line added after 64th character, execute: sed -i -e "s/.\{64\}/&\n/g" trustedCert_x.crt Verify the backed up certificate: openssl x509 -text -in trustedCert_x.crt Proceed to unpublish certificate from vmdir if the backup was successful: /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert trustedCert_x.crt Initiate VECS store refresh to delete the unpublished certificate from the VECS store /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh Publish the backed trusted root certificates back to vmdir /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert trustedCert_x.crt Verify if the certificate has been added: /usr/lib/vmware-vmafd/bin/dir-cli trustedcert list Initiate VECS store refresh to add the published certificate to vecs. /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh Verify VECS TRUSTED_ROOTS store entries: /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text