Replacing the vCenter server certificate through SDDC Manager fails for "Exception found (AFD Native Error Occured: 11)"

Products

VMware Cloud Foundation

Issue/Introduction

Replacing the vCenter certificate through SDDC Manager fails with below error
```
Exception found (AFD Native Error Occured: 11)
```
If the vCenter certificate replacement fails or the SDDC Manager and vCenter re-trust fails the vCenter is unable to process any certificate related workflows, which can lead other workflows like add/ remove host, etc to fail.

Attempting to the vCenter certificate replacement workflow fails with below error in the /var/log/vmware/vcf/operationsmanager/operationsmanager.log

YYYY-MM-DDTHH:MM:SS.287+0000 ERROR [vcf_om,6c92de84a35548da,391e] [c.v.v.c.vc.VCenterCertificatePlugin,om-exec-29] Replace certificate failed for resource: {"version":"7.0.2.00000-17694817","hostName":"<FQDN>","id":"******03-2**2-4**4-b**1-d******4*3","vmName":"<VM_NAME>","resourceType":"vcenter","credentials":[{"username":"root","secret":"*****","type":"ssh"},{"username":"[email protected]","secret":"*****","type":"sso"}],"ipAddress":"<IP>","sans":["<FQDN>"],"master":false} : 500 Internal Server Error: [{"type":"com.vmware.vapi.std.errors.error","value":{"error_type":"ERROR","messages":[{"args":["AFD Native Error Occured: 11"],"default_message":"Exception found (AFD Native Error Occured: 11)","id":"com.vmware.certificatemanagement.error"}]}}] : {}com.vmware.vcf.certmgmt.exceptions.CertificateReplacementException: 500 Internal Server Error: [{"type":"com.vmware.vapi.std.errors.error","value":{"error_type":"ERROR","messages":[{"args":["AFD Native Error Occured: 11"],"default_message":"Exception found (AFD Native Error Occured: 11)","id":"com.vmware.certificatemanagement.error"}]}}]

The SDDC Manager and vCenter re-trust sub task fails with below error in /var/log/vmware/vcf/operationsmanager/operationsmanager.log

YYYY-MM-DDTHH:MM:SS.959+0000 ERROR [vcf_om,6c92de84a35548da,701f] [c.v.v.c.v.VCenterCertificatePluginService,om-exec-25] 500 Internal Server Error: [{"type":"com.vmware.vapi.std.errors.internal_server_error","value":{"error_type":"INTERNAL_SERVER_ERROR","messages":[{"args":["com.vmware.vapi.std.errors.Error"],"default_message":"Provider method imp... (482 bytes)]
org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 Internal Server Error: [{"type":"com.vmware.vapi.std.errors.internal_server_error","value":{"error_type":"INTERNAL_SERVER_ERROR","messages":[{"args":["com.vmware.vapi.std.errors.Error"],"default_message":"Provider method imp... (482 bytes)]
 at org.springframework.web.client.HttpServerErrorException.create(HttpServerErrorException.java:100)
 at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:186)
 at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:125)
 at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
 at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:780)
 at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:738)
 at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:712)
 at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:600)

Running certificate force refresh on the vCenter reports the following error

/usr/lib/vmware-vmafd/bin/vecs-cli force-refresh

Error: Failed to trigger root cert refresh
vecs-cli failed. Error 11: Possible errors:
LDAP error: Administrative limit exceeded
Win Error: Operation failed with error ERROR_BAD_FORMAT (11)

Adding a trusted root certificate to Certificate Management via UI fails with error
```
Error occurred while adding trusted root certificates
```

Environment

VMware Cloud Foundation 4.x

Cause

This issue occurs if root certificate content is of length in the order of 233 multiples.

Resolution

This is a known issue affecting all VCF 4.x releases and will be resolved in a future release.

To workaround this problem, please either use the attached fixCertRetrust.sh script or follow the steps below in Option 2.

Option 1:

Download the fixCertRetrust.sh script.
Using a file transfer utility, like WinSCP, copy the file to /tmp/ on the the vCenter appliance.
Open an SSH session to the vCenter appliance/
Change the permissions on the file
```
chmod +x fixCertRetrust.sh
```
Run the script
```
sh fixCertRetrust.sh
```

Option 2:

Open an SSH session to the vCenter appliance with the root user.

Get the trusted root certificates list

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert list --login administrator@<domain_name>

Sample outout

Number of certificates: 1
#1:
CN(id):         xxxxxxxxxxxxxxxxxxxxxxxxDB56
Subject DN:     CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=vcsa01.example.com, OU=VMware Engineering
CRL present:    yes

Take a backup of all the certificates; using the IDs the output from step 2

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert get --id <ID> --outcert trustedCert_x.crt

Sample output

root@vcsa01 [ /etc/vmware-syslog ]# /usr/lib/vmware-vmafd/bin/dir-cli trustedcert get --id xxxxxxxxxxxxxxxxxxxxxxxDB56 --outcert trustedCert_x.crt
Enter password for [email protected]:
Certificate retrieved successfully

If there is no new line added after 64th character, execute the below command
```
sed -i -e "s/.\{64\}/&\n/g" trustedCert_x.crt
```

Verify the backed up certificate:

openssl x509 -text -in trustedCert_x.crt

Proceed to unpublish certificate from vmdir if the backup was successful

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert trustedCert_x.crt

Initiate VECS store refresh to delete the unpublished certificate from the VECS store
```
/usr/lib/vmware-vmafd/bin/vecs-cli force-refresh
```

Publish the backed trusted root certificates back to vmdir

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert trustedCert_x.crt

Verify if the certificate has been added

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert list

Initiate VECS store refresh to add the published certificate to vecs.
```
/usr/lib/vmware-vmafd/bin/vecs-cli force-refresh
```

Verify VECS TRUSTED_ROOTS store entries

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text

Attachments

fixCertRetrust get_app