VLCM page on Vcenter web UI shows error: An Unexpected error occured

Products

VMware vCenter Server

Issue/Introduction

vCenter UI is accessible however, when trying to browse to the lifecycle manager landing page a red banner is present and unable to access VLCM page options.

'An unexpected error has occurred' and 'you have no privileges to view this object or it does not exist'
In /var/log/vmware/vmware-updatemgr/vum-server/vmware-vumserver.log, you see the following errors:
- Failed to get Saml token after refreshing STS certs error: SSL Exception: Verification parameters: certificate has expired
- N7Vmacore3Ss118SSLVerifyExceptionE (SSL Exception: Verification parameters: -> PeerThumbprint: ... -> The remote host certificate has these problems: * certificate has expired)
- When checking services running in an SSH to vCenter multiple services are in a stopped state

service-control --status --all

Stopped:

applmgmt observability-vapi vmcam vmonapi vmware-certificatemanagement vmware-hvc vmware-imagebuilder vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-sps vmware-topologysvc vmware-vcha vmware-vpxd-svcs vmware-vsan-health vstats

Environment

vCenter 8.x

Cause

This issue is observed when the Machine SSL certificate has expired, preventing a number of services to start as expected.

The vmware-vum-server service fails to authenticate against the Security Token Service (STS) and VMware Lookup Service because expired SSL certificates (specifically STS or Solution User certificates) are retained in the local trust store.

Resolution

Validate the certificate status:

In an ssh session to vCenter run the following command to validate the 'Not After' date of the certificates

for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

If the Machine SSL certificate has expired, proceed to replace this following the steps below.

1. Take snapshot of vCenter

If this vCenter is part of Enhanced Linked Mode the use of offline snapshots in ELM deployments is very strongly recommended for a safe rollback point. This means all appliances should be gracefully shut down, and snapshots need to be taken while the VCSAs are in powered off state (at the same time).

If any change must be reverted, restore all of the nodes in the ELM deployment to this offline/consistent snapshot state

For further information related to vCenter backup/snapshot best practices, refer to VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice.

2. Download the vCert tool from KB

- Follow the steps in the KB to upload on to the vCenter appliance

- Browse into the vCert directory

- Execute the script ./vCert.py

- When prompted replace the Machine SSL using the following options

Scenario 1: To Replace Machine SSL certificate with a VMCA-signed certificate:

=>3. Manage certificates
=>1. Machine SSL certificate
=>1. Replace Machine SSL certificate with a VMCA-signed certificate

Scenario 2:To Replace Machine SSL certificate with a custom CA-signed certificate if using 3rd party CA certificate :

=> 2. Replace Machine SSL certificate with a custom CA-signed certificate if using 3rd party CA certificate

3. When prompted restart services.

Scenario 3:If all certificates are to be renewed as per the events in vmserver.log:

The vmware-vum-server service fails to authenticate against the Security Token Service (STS) and VMware Lookup Service because expired SSL certificates (specifically STS or Solution User certificates) are retained in the local trust store.

To resolve this issue, you must renew the expired certificates using the vCert utility.

Use option 6
To reset all certificates along with STS certificates

If the issue persists, contact Broadcom Support

Additional Information

See KB vCenter Machine SSL Certificate Replacement with Custom CA-Signed Certificate via vSphere Client UI if using a CA-signed certificate