Replacing the vCenter certificate through SDDC Manager fails for "Exception found (AFD Native Error Occured: 11)"

Products

• VMware Cloud Foundation

Issue/Introduction

If the vCenter certificate replacement fails or the SDDC Manager and vCenter re-trust fails the  vCenter is unable to process any certificate related workflows, which can lead other workflows like add/ remove host, etc to fail.

Symptoms:

• Attempting to the vCenter certificate replacement workflow fails with below error in the /var/log/vmware/vcf/operationsmanager/operationsmanager.log:

2021-05-11T05:07:00.287+0000 ERROR [vcf_om,6c92de84a35548da,391e] [c.v.v.c.vc.VCenterCertificatePlugin,om-exec-29] Replace certificate failed for resource: {"version":"7.0.2.00000-17694817","hostName":"<FQDN>","id":"54df8203-2e72-40b4-ba51-d1dc4af443e3","vmName":"<VM_NAME>","resourceType":"vcenter","credentials":[{"username":"root","secret":"*****","type":"ssh"},{"username":"[email protected]","secret":"*****","type":"sso"}],"ipAddress":"<IP>","sans":["<FQDN>"],"master":false} : 500 Internal Server Error: [{"type":"com.vmware.vapi.std.errors.error","value":{"error_type":"ERROR","messages":[{"args":["AFD Native Error Occured: 11"],"default_message":"Exception found (AFD Native Error Occured: 11)","id":"com.vmware.certificatemanagement.error"}]}}] : {}com.vmware.vcf.certmgmt.exceptions.CertificateReplacementException: 500 Internal Server Error: [{"type":"com.vmware.vapi.std.errors.error","value":{"error_type":"ERROR","messages":[{"args":["AFD Native Error Occured: 11"],"default_message":"Exception found (AFD Native Error Occured: 11)","id":"com.vmware.certificatemanagement.error"}]}}]

at com.vmware.vcf.certmgmt.vc.VCenterCertificatePluginService.replaceCertificate(VCenterCertificatePluginService.java:230)

at com.vmware.vcf.certmgmt.vc.VCenterCertificatePluginService.uploadCertificate(VCenterCertificatePluginService.java:179)
 at com.vmware.vcf.certmgmt.vc.VCenterCertificatePlugin.replaceCertificate(VCenterCertificatePlugin.java:83)
at com.vmware.vcf.certmgmt.vc.VCenterCertificatePlugin$$FastClassBySpringCGLIB$$874f9739.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56)

• The SDDC Manager  and vCenter re-trust sub task fails with below error:

2021-05-11T05:15:22.959+0000 ERROR [vcf_om,6c92de84a35548da,701f] [c.v.v.c.v.VCenterCertificatePluginService,om-exec-25] 500 Internal Server Error: [{"type":"com.vmware.vapi.std.errors.internal_server_error","value":{"error_type":"INTERNAL_SERVER_ERROR","messages":[{"args":["com.vmware.vapi.std.errors.Error"],"default_message":"Provider method imp... (482 bytes)]
org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 Internal Server Error: [{"type":"com.vmware.vapi.std.errors.internal_server_error","value":{"error_type":"INTERNAL_SERVER_ERROR","messages":[{"args":["com.vmware.vapi.std.errors.Error"],"default_message":"Provider method imp... (482 bytes)]
        at org.springframework.web.client.HttpServerErrorException.create(HttpServerErrorException.java:100)
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:186)
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:125)
        at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
        at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:780)
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:738)
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:712)
        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:600)
 

• Running /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh on the vCenter reports the following

Error: Failed to trigger root cert refresh
vecs-cli failed. Error 11: Possible errors:
LDAP error: Administrative limit exceeded
Win Error: Operation failed with error ERROR_BAD_FORMAT (11)
 

• Adding a trusted root certificate to Certificate Management via UI fails with "Error occurred while adding trusted root certificates"

 

Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Foundation 4.x

Cause

This issue occurs if root certificate content is of length in the order of 233 multiples. 

Resolution

This is a known issue affecting all VCF 4.x releases and will be resolved in a future release. 

Workaround:
To workaround this problem, please either use the attached fixCertRetrust.sh script or follow the steps below in Option 2.

Option 1:

1. Download the fixCertRetrust.sh script.
2. Using a file transfer utility, like WinSCP, copy the file to /tmp/ on the the vCenter appliance.
3. Open an SSH session to the vCenter appliance/
4. Change the permssions on the file:

chmod +x fixCertRetrust.sh

1. Run the script:

sh CertRepublish.sh

Option 2:

1. Open an SSH session to the vCenter appliance with the root user. 
2. Get the trusted root certificates list 

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert list --login administrator@<domain>

3. Take a backup of all the certificates; using the IDs the output from step 1

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert get --id <ID> --outcert trustedCert_x.crt

4. If there is no new line added after 64th character, execute:

sed -i -e "s/.\{64\}/&\n/g"  trustedCert_x.crt

5. Verify the backed up certificate:

openssl x509 -text -in  trustedCert_x.crt

6. Proceed to unpublish certificate from vmdir if the backup was successful:

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert trustedCert_x.crt

7.  Initiate VECS store refresh to delete the unpublished certificate from the VECS store

 /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh

8.  Publish the backed trusted root certificates back to vmdir

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert trustedCert_x.crt 

9. Verify if the certificate has been added:

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert list

10. Initiate VECS store refresh to add the published certificate to vecs. 

/usr/lib/vmware-vmafd/bin/vecs-cli force-refresh

11. Verify VECS TRUSTED_ROOTS  store entries:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text