vCenter topologysvc fails to start after a reboot

Products

• VMware vCenter Server

Issue/Introduction

• The vmware-topologysvc fails to start on vCenter Server.
• Kubernetes persistent volumes using the VMware CSI driver are non-functional (503 errors).
• The Cloud Native Storage (CNS) or Cloud Native Container tab is missing or failing to load in the vSphere Client.
• In the /var/log/vmware/vmon/vmon.log:
  YYY-MM-DDTHH:MM:SS.MSSZ Wa(03) host-#### <topologysvc> Service pre-start command's stderr: self._sslobj.do_handshake()
YYY-MM-DDTHH:MM:SS.MSSZ Wa(03)+ host-#### ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1017)
YYY-MM-DDTHH:MM:SS.MSSZ Er(02) host-#### <topologysvc> Service pre-start command failed with exit code 1.
YYY-MM-DDTHH:MM:SS.MSSZ Wa(03) host-#### [ReadSvcSubStartupData] No startup information from topologysvc.

Cause

This issue occurs when an expired Trusted Root or Intermediary certificate remains in the TRUSTED_ROOTS VECS store and the VMware Directory (vmdir).

Resolution

Ensure a powered-off snapshot of the vCenter Server (and all nodes in the SSO domain if in ELM) is taken before proceeding Best practices for using VMware snapshots in the vSphere environment

1. SSH to the vCenter Server as root
2. Identify the Expired Certificate in the Trusted Roots store: /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | grep -E "Alias|Not After|Subject Key Identifier"
3. Remediate using vCert vCert - Scripted vCenter expired certificate replacement
   1. Navigate to Option 3 (Manage Certificates) > Option 3 (CA certificates in VMware Directory).
   2. Select Option 2 (Remove CA certificate(s) from VMware Directory).
   3. Select the numeric ID corresponding to the expired certificate identified in the previous steps.
4. Force a refresh of the local VECS store: /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh
5. Verify and Restart Services
   1. Start all services: service-control --start --all
   2. Confirm topologysvc is running: service-control --status vmware-topologysvc