Services in vCenter on VMC may in stopped state due to expired HLM certificate

Products

VMware Cloud on AWS

Issue/Introduction

You may experience these symptoms.

Multiple services on the vCenter hosted in VMC infra might be in a stopped state.

Running:
 lookupsvc lwsmd pschealth vlcm vmafdd vmcad vmdird vmonapi vmware-analytics vmware-cis-license vmware-eam vmware-envoy vmware-envoy-hgw vmware-envoy-sidecar vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-stsd vmware-trustmanagement vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vsm vtsdb
StartPending:
 vmware-sps vmware-vpxd-svcs vmware-vsan-health
Stopped:
 applmgmt observability observability-vapi vmcam vmware-certificateauthority vmware-certificatemanagement vmware-content-library vmware-hvc vmware-imagebuilder vmware-infraprofile vmware-netdumper vmware-perfcharts vmware-pod vmware-rbd-watchdog vmware-topologysvc vmware-updatemgr vmware-vcha vmware-vdtc vsphere-ui vstats wcp

Manually starting the services might fail.
Checking the certificates in the VECS store using the command below may show no expired certificates or may show one expired certificate in TRUSTED_ROOTS.

for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

Sample output:

You remove the expired cert from the TRUSTED_ROOTS using KB:Verify and remove CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store(VECS)
The vCert script may no longer list any expired certificates.
Even after all the cert checks, the vCenter services are not starting.
If you take the log bundle of vCenter, you will notice one expired certificate in sts-certificates.txt, stored in the commands folder.

Sample output:
dn: cn=SigningCertificateChain0,cn=TrustedCertificateChains,cn=116d5f9e-xxxx-xxxx-xxxx-xxxxxxxxxxxx ,cn=VCTrusts,cn=vmc.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vmc,dc=local
Certificate 1
Certificate will expire
        Issuer: CN=CA, CN=example.com, dc=vsphere,dc=local, C=US
        Validity
            Not Before: Jan  4 18:19:09 2016 GMT
            Not After : Jan  1 18:19:09 2026 GMT
        Subject: CN=ssoserverSign,dc=vsphere,dc=local, C=US
Certificate 2
Certificate will expire
        Issuer: CN=CA, CN=example.com, dc=vsphere,dc=local, C=US
        Validity
            Not Before: Jan  4 18:19:07 2016 GMT
            Not After : Jan  1 18:19:07 2026 GMT
        Subject: CN=CA, CN=example.com, dc=vsphere,dc=local, C=US

Environment

vCenter on VMC

Cause

The issue is caused due to expired VCTrust certificates introduced when the HLM integration happens.

This certificate is stored in the vmdird particular VMware Directory branch (cn=VCTrusts,cn=vmc.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vmc,dc=local)

Resolution

There are two ways to resolve this issue.

Note: Please ensure that you have a proper snapshot or backup of the vCenter before implementing any of the solutions.

Solution:1

Use JXplorer to connect to vmdir. (Ref KB Using JXplorer to connect to the vSphere Single Sign-on)
Expand Local > vSphere > Services
Expand IdentityManager > Tenants > vsphere.local
Select TrustedCertificateChains and expand this.
Here you will see a SigningCertificateChain0 with cn=<number-value>.
Select this object and delete it.
Restarted the vCenter services

root@vcenter [ ~ ]# service-control --stop --all <------- first, stop all services

root@vcenter [ ~ ]# service-control --start --all <----- afterwards, start all services

Solution:2

Run the following command to execute an ldapsearch on the vSphere SSO domain

/opt/likewise/bin/ldapsearch -b "cn=VCTrusts,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=sphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W

or

/opt/likewise/bin/ldapsearch -x -h localhost -p 389 \
  -D "cn=Administrator,cn=Users,dc=vmc,dc=local" -W \
  -b "cn=VCTrusts,cn=vmc.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vmc,dc=local" \
  -s sub "(cn=SigningCertificateChain*)" dn

Note: If you have the proper CN number from sts-certificates.txt, then you can specify the CN name in the ldapsearch command.

Example: dn: cn=SigningCertificateChain0,cn=TrustedCertificateChains,cn=116d5f9e-xxxx-xxxx-xxxx-xxxxxxxxxxxx ,cn=VCTrusts,cn=vmc.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vmc,dc=local

/opt/likewise/bin/ldapsearch -x -h localhost -p 389 \
  -D "cn=Administrator,cn=Users,dc=vmc,dc=local" -W \
  -b "cn=SigningCertificateChain0,cn=TrustedCertificateChains,cn=116d5f9e-xxxx-xxxx-xxxx-xxxxxxxxxxxx,cn=VCTrusts,cn=vmc.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vmc,dc=local" \
  -s base

Once you have the CN and certificate information validated. We will use the ldapdelete command to delete the object.

/opt/likewise/bin/ldapdelete -h localhost -D "cn=Administrator,cn=Users,dc=vmc,dc=local" -W "<DN (distinguished name)>" -r -v

or

/opt/likewise/bin/ldapdelete -x -h localhost -p 389 \
  -D "cn=Administrator,cn=Users,dc=vmc,dc=local" -W \
  "cn=SigningCertificateChain0,cn=TrustedCertificateChains,cn=116d5f9e-xxxx-xxxx-xxxx-xxxxxxxxxxxx,cn=VCTrusts,cn=vmc.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vmc,dc=local"

Restarted the vCenter services

root@vcenter [ ~ ]# service-control --stop --all <------- first, stop all services

root@vcenter [ ~ ]# service-control --start --all <----- afterwards, start all services