vCenter Server not visible in Enhanced Linked Mode inventory due to expired Machine SSL certificate

Products

• VMware vCenter Server

Issue/Introduction

• One or more vCenter Servers fail to appear in the Enhanced Linked Mode (ELM) configuration, where other vCenter Servers may be visible.
• Checking the affected vCenter Server reveals that multiple vCenter services are stopped and failing to start, such as:
  vmware-certificatemanagement, vmware-vpxd-svcs, vmware-topologysvc, vmware-vsan-health, vmware-hvc, vmware-sps, and vstats
  
  
• The affected vCenter server does not show up in the vCenter server inventory even though it is in enhanced linked mode.
• The vCenter UI shows the below error:
  Could not connect to one or more vCenter Server systems: https://<vCenter_fqdn>:443/sdk


• The below error is logged in the /var/log/vmware/vmon/vmon.log:
  File "/usr/lib/python3.7/ssl.py", line ##, in do_handshake
  self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:###)
• Running for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done; shows an expired Not After date for the __MACHINE_CERT alias.

Environment

• VMware vCenter Server 7.x
• VMware vCenter Server 8.x

Cause

The Machine SSL certificate on one or more vCenter Server nodes has expired. In an ELM environment, this prevents the lookup service and vSphere Client from authenticating with the remote node.

Resolution

1. Take offline (powered-off) snapshots of all vCenter Server nodes in the SSO domain Snapshot Best practices for vCenter Server Virtual Machines
2. Log in to the affected vCenter Server via SSH as root.
3. Download and run the vCert utility vCert - Scripted vCenter expired certificate replacement
4. Select Option 3 (Manage certificates) and then Option 1 (Machine SSL).
5. Select Option 1 (Replace with VMCA-signed certificate) to restore immediate service connectivity.
6. Restart all services using the command: service-control --stop --all && service-control --start --all
7. Verify ELM connectivity in the vSphere Client.

Additional Information

Note: If the environment requires Custom CA-signed certificates, use the vSphere Client UI (Home > Administration > Certificate Management) to generate a new CSR and import the new Custom CA certificate once UI access is restored vCenter Machine SSL Certificate Replacement with Custom CA-Signed Certificate via vSphere Client UI