Could not connect to one or more vCenter Server system after renewing machine_ssl_cert due to expired intermediary certificate

Products

• VMware vCenter Server

Issue/Introduction

vpxd-svcs failing to start with no logging in /var/log/vmware/vpxd-svcs/vpxd-svcs.log

Errors logged in /var/log/vmware/vmon/vmon.log for vpxd-svcs:

<timestamp> Wa(03) host-##### <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):
<timestamp> Wa(03)+ host-#####   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 100, in <module>
<timestamp> Wa(03)+ host-#####     endpoint_registration_runner(logging_file)
<timestamp> Wa(03)+ host-#####   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 65, in endpoint_registration_runner
<timestamp> Wa(03)+ host-#####     UpdateTaggingServiceGrpcEndpoint(logging_file).run()

.....

<timestamp> Wa(03)+ host-#####   File "/usr/lib/python3.7/ssl.py", line 1168, in do_handshake
<timestamp> Wa(03)+ host-#####     self._sslobj.do_handshake()
<timestamp> Wa(03)+ host-##### ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076)
<timestamp> Wa(03)+ host-#####
<timestamp> Er(02) host-##### <vpxd-svcs> Service pre-start command failed with exit code 1.

Environment

VC 7.x

VC 8.x

Cause

Intermediary certificate which directly signs machine_ssl_cert is expired

Even if renewing machine_ssl_cert with a valid intermediary cert, if it has the same attributes as the one currently published in trusted_roots on the vCenter - it will not be updated during renewal of machine_ssl_cert 

Resolution

Manually unpublish the expired intermediary certificate and publish the new one

To find the intermediary certificate's alias:

>View the machine_ssl_cert and note down either Issuer or Authority Key Identifier:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store machine_ssl_cert --text

Example with only interest fields:

Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=intermediary.com, DC=vsphere, DC=local, C=US, ST=California, O=intermediary.com, OU=VMware Engineering
        Validity
            Not Before: Jan 1 00:00:00 2000 GMT
            Not After : Jan 3 00:00:00 2035 GMT
        Subject: CN=myvcenter.com, DC=vsphere, DC=local, C=US, ST=California, O=myvcenter.com, OU=VMware Engineering

...

 X509v3 extensions:
            X509v3 Authority Key Identifier:
                6F:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:0D

 

>Search trusted_roots for a cert with its Subject or Subject Key Identifier the same as the noted Issuer or Authority Key Identifier respectively. Note down the Alias of the matching cert:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store trusted_roots --text

 

>Once the Alias has been noted down - follow steps 2 to 7 from KB Verify and remove CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store(VECS) to remove the expired intermediary

 

>Upload a copy of the new intermediary in crt or cer format to the vCenter and run the below command to publish:

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert <PATH_TO_CERT>

 

>Restart all services to apply the change:

service-control --stop --all && service-control --start --all

 

Note: If machine_ssl_cert is also expired it will need to be renewed before restart of services