vCenter custom Machine_SSL certificate fails with error "[CERTIFICATE] Replace cert Failed: Exception found (Failed to notify APPLMGMT on http://localhost:1080/api/appliance/certificates/notification, on all retries.)"

Products

• VMware vCenter Server 8.0

Issue/Introduction

Renewing Machine_SSL custom cert via vCenter UI fails while importing signed certificate

Environment

vCenter 8.0

Cause

The old expired trusted root cert subject key is still used as authority key in Machine_SSL while creating a CSR
After new Root Certificate is generated, the old one needs to be unpublished from the vmdir and VECS store
When checking if the Root certificate is signing any other certificate in the environment make sure to note below:

                 /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | grep -i alias

• • Old Trusted Root Certificate Subject Key Identifier

    12:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:21
    
    

  • Compare the Subject Key Identifier from the old Root Certificate above with certificates from all stores such as Machine SSL, machine, vpxd etc and see if it matches with their Authority Key Identifier
    

    /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | less

  • In the example below we see the Authority Key Identifier is not matching with the Subject Key Identifier of the Root Certificate, meaning the vpxd certificate below is not signed with the Old Root Certificate. Proceed to check for all stores 

    vpxd
    
     X509v3 Subject Key Identifier:
                    ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
                X509v3 Authority Key Identifier:
                    45:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:54

Resolution

1. Take a snapshot of the vCenter. Offline snapshots if linked mode is enabled, online snapshots if there is only one vCenter and no linked mode in place. Read more at Snapshot Best practices for vCenter Server Virtual Machines
2. Once snapshots are completed proceed to unpublish the expired trusted root certificate. Refer to Verify and remove CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store(VECS), or https://knowledge.broadcom.com/external/article/319476
3. Generate new CSR and get it signed
4. Import and replace certificate via vCenter UI using embedded key option