vmware-vpxd service fails to start with vpxd.log indicating "Invalid certificate string"

Products

• VMware vCenter Server

Issue/Introduction

• vmware-vpxd service fails to start.
• Log snippets of /var/log/vmware/vpxd/vpxd.log states "invalid certificate string" due to invalid base64 encoded string for an expired certificate in the trusted roots store during CheckCertificateExpiry. The certificate is in Base64 encoded but the encoded string is found to be invalid.

YYYY-MM-DDTHH:MM error vpxd[04524] [Originator@6876 sub=Main opID=CheckCertificateExpiry-7460ee5a] Invalid certificate string [MIIIN#########################################AQsF
--> ADBlM######################################################hcmVu
--> YTEU######################################################RlcnBy
 Error:N7Vmacore21Base64FormatExceptionE(Invalid base64-encoded string)

Cause

• vpxd encounters the N7Vmacore21Base64FormatExceptionE(Invalid base64-encoded string) error while trying to read a certificate in the trusted roots store and it cannot properly decode the certificate as the certificate string passed to the Base64 decoder contained non-Base64 characters.

Resolution

• Delete the invalid certificate from the trusted roots store by running the following commands:

Note: Before the troubleshooting, you must take backup of the vCenter Server Appliance. If the vCenter Servers are in Enhanced Linked Mode (ELM), make sure to take offline snapshot of all the vCenter Servers that are participating in the Enhanced Linked Mode.

Reference article: VMware vCenter in Enhanced Linked Mode pre-changes snapshot best practice (313886)

1. Connect SSH to the vCenter Server Appliance.

2. List the certificates using vecs-cli.

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less

3. Find the expired certificate that needs to be removed and make a note of the Alias.

Alias : 12ad456faa####################d101ad59df

4. Using the Alias ID located in Step 3, run the following command to save the certificate to /root/ folder, adjusting appropriately for the environment:

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias 12ad456faa####################d101ad59df --output /root/<aliasID>.cer

5. Unpublish the expired/expiring CA certificate from VMDIR, it will prompt for SSO Administrator credentials.

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /root/<aliasID>.cer

6. Delete the certificate from VECS utilizing the Alias located in Step 2:

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias 212ad456faa####################d101ad59df

7. Perform force refresh of VECS to sync certificate from VMDIR.

/usr/lib/vmware-vmafd/bin/vecs-cli force-refresh

8. Confirm that the certificate is no longer present.

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | grep Alias

9. Restart all services on the vCenter Servers, ensuring that all services start and respond normally, and that login and management of the environment are functioning properly.

service-control --stop --all && service-control --start --all

Additional Information

For information in to supported certificate format, refer: vSphere Certificate Requirements for Different Solution Paths