Error 503 when trying to login to vami, trusted roots CA certificate expired

Products

VMware vCenter Server

Issue/Introduction

Error 503 on the vCenter Appliance Management https://vcenter_url:5480 and is not accessible to login
Services applmgmt, certificatemanagement and certificateauthority will not start (service-control --start <servicename>)
Unable to manage certificates in vSphere Client -> Administration -> Certificates -> Certificate Management -> Trusted Root
vCenter - /var/log/vmware/vpxd/vpxd.log will show the following message for the expired CA certificate
[YYYY-MM-DDTHH:MM:SS] warning vpxd[30469] [Originator@6876 sub=Main opID=CheckCertificateExpiry-57e82b11] Certificate [Subject: <Certificate Subject>] from store <VECS Store Name> will expire on YYYY-MM-DD HH:MM:SS

Environment

vCenter 7.x

vCenter 8.x

Cause

A needed Intermediate CA Certificate in vCenter Trusted Roots has expired.

Resolution

It is required to backup and snapshot vCenter prior to performing these steps

Review the certificates in the trusted store.

vcsa# for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

Check vpxd log for the expired cert name.

vcsa# grep -i expiry /var/log/vmware/vpxd/vpxd.log

Check dir-cli certificates against vecs-cli local certificates to locate the expired certificate.

vcsa# /usr/lib/vmware-vmafd/bin/dir-cli trustedcert list

vcsa# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less

Note the SKI ID of the expired certificate to export and unpublish the expired certificate.

vcsa# /usr/lib/vmware-vmafd/bin/dir-cli trustedcert get --id ################ --outcert /root/oldcert.cer

vcsa# openssl x509 -in /root/oldcert.cer -text -noout

vcsa# /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /root/oldcert.cer

vcsa# /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias ################

Get the base64 hash of the correct intermediate CA certificate to publish it.

vcsa# openssl s_client -connect intermediateCA_name:443 -showcerts

Note: When snipping text, include the BEGIN and END lines for the certificate and paste in a new file /root/newcert.cer

-----BEGIN CERTIFICATE-----
MIIFyjCCBLKgAwIBAgIKYURFHAAAAAAABDANBgkqhkiG9w0BAQUFADBCMRMwEQYK
..........
...snip...
..........
TmqX6OuznopBJKNW5Z5LbHzuUCfY8ryBhYZhHKsf9CmZa12j/ODfznFtAgbPNw==
-----END CERTIFICATE-----

vcsa# openssl x509 -in /root/newcert.cer -text -noout

vcsa# /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /root/newcert.cer

Refresh the local vecs trusted roots store.

vcsa# /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh

Validated new Intermediate CA certificate is published.

vcsa# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less

vcsa# /usr/lib/vmware-vmafd/bin/dir-cli trustedcert list

vcsa# /usr/lib/vmware-vmafd/bin/dir-cli trustedcert get --id ################ --outcert /root/checkcert.cer

vcsa# openssl x509 -in /root/checkcert.cer -text -noout

Restart services.

vcsa# service-control --stop --all; vcsa# service-control --start --all

Check that certtificatemanagement, certificateauthority and applmgmt services come up.

vcsa# service-control --status

Ensure vami is accessible again to login

https://vcenter_url:5480