vCenter Server inaccessible due to expired vCenter certificates

Products

• VMware vCenter Server

Issue/Introduction

• The following error is observed while accessing vSphere Client:
  
  • [500] An error occurred while fetching identity providers. Try again. If problem persists, contact your administrator. Back to login screen.
• vSphere Client is inaccessible and shows following message: "no healthy upstream".
• Running "service-control --status --all" in vCenter SSH confirms multiple services are in a Stopped or StartPending state, including vmware-vpxd and vmware-vpxd-svcs.
• Attempting to start the vmware-vpxd service using the command "service-control --start vmware-vpxd" results in the following error:

Performing start operation on service vpxd...
Error executing start on service vpxd. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd"
            ],
            "localized": "An error occurred while starting service 'vpxd'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null

• Validating the vCenter certificates using the following shows Machine_SSL and/or Solution user certificates are expired.
  
  for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

[*] Store : machine
Alias : machine
        Not After : Month DD HH:MM:SS YYYY GMT
[*] Store : vsphere-webclient
Alias : vsphere-webclient
        Not After : Month DD HH:MM:SS YYYY GMT
[*] Store : vpxd
Alias : vpxd
        Not After : Month DD HH:MM:SS YYYY GMT
[*] Store : vpxd-extension
Alias : vpxd-extension
        Not After : Month DD HH:MM:SS YYYY GMT
  
  
• In the /var/log/vmware/vpxd/vpxd.log, reports the vCenter Machine_SSL certificate had expired:

YYYY-MM-DDTHH:MM info vpxd[#####] [Originator@#### sub=IO. Http] Set user agent error; state: 1, (null), N7Vmacore3Ss118SSLVerify(SSL Exception: Verification parameters:
-> PeerThumbprint: ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
-> ExpectedThumbprint:
-> ExpectedPeerName: <vCenter FQDN>
-> The remote host certificate has these problems:
-> 
-> * certificate has expired)

Environment

• vCenter 8.x

Cause

This issue occurs because the vCenter Server Solution User certificates (machine, vsphere-webclient, vmware-vpxd, vpxd-extension, and hvc) and/or Machine_SSL certificate has expired. These internal certificates need to be renewed periodically, and are required for vCenter services to authenticate with the Security Token Service (STS) to acquire SAML tokens. 

Resolution

To resolve the issue, renew the vCenter Server Solution User certificates using the vCert script with VMCA as the certificate authority. Follow the detailed steps outlined in the KB article: vCert - Scripted vCenter expired certificate replacement.

1. Take an offline snapshot of vCenter servers before proceeding. If the vCenter is in linked mode, take offline snapshot of all the vCenter Servers participating in the Enhanced linked mode. For more information refer to VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice.
2. Download the vCert tool and upload it to the vCenter's /root or /tmp directory. vCert script can be downloaded in the following KB: vCert - Scripted vCenter expired certificate replacement.
3. Unzip the vCert tool: 
     unzip -q vCert-########.zip
4. Navigate to the new vCert directory: 
     cd vCert-########.zip
5. Give full permission to the vCert.py script:
     chmod +x vCert.py
6. Run the vCert tool: 
     ./vCert.py
7. Acknowledge risk to continue: Do you acknowledge the risks and wish to continue? [y/n]: y
8. Depending on the certificates expired, follow the next steps:
   1. If only Solution user certificates are expired, renew Solution user certificates using the following options:
      1. Select option 3 - Manage certificates, to manage the certificates. 
      2. Select option 2 - Solution User certificates to renew Solution user certificates.
      3. Select option 1 - Replace with VMCA signed certificates
      4. Restart services when prompted.
   2. If only Machine_SSL certificate is expired, renew Machine_SSL certificate using the following options:
      1. Select option 3 - Manage certificates, to manage the certificates. 
      2. Select option 1 - Machine SSL certificate to renew Machine_SSL certificate.
      3. Select option 1 - Replace with VMCA signed certificates
      4. Restart services when prompted.
   3. If both Machine_SSL and Solution user certificates are expired, renew Solution user certificates first and then proceed to reset Machine_SSL certificate:
      1. Select option 3 - Manage certificates, to manage the certificates.
      2. Select option 2 - Solution User certificates to renew Solution user certificates.
      3. Select option 1 - Replace with VMCA signed certificates
      4. Do not restart the services when prompted. Return to Manage Certificates menu.
      5. Select option 3 - Manage certificates, to manage certificates.
      6. Select option 1 - Machine SSL certificate to renew Machine_SSL certificate.
      7. Select option 1 - Replace with VMCA signed certificates
      8. Restart services when prompted.