vCenter Web Client stuck at spinning wheel after login due to Expired Certificates

Products

VMware vCenter Server

Issue/Introduction

After entering credentials into the vSphere Web Client, the interface remains stuck on a loading screen (spinning wheel). This affects both the [email protected] account and Active Directory (AD) integrated accounts.
The vCenter Server Appliance Management Interface (VAMI) at https://<vcenter_fqdn>:5480 is accessible.
vCenter is accessible via SSH.

Disk space usage (df -h) is normal (no partitions are at 100%).

Sample output

Filesystem                                   Size  Used Avail Use% Mounted on
devtmpfs                                     4.0M     0  4.0M   0% /dev
tmpfs                                         15G 1004K   15G   1% /dev/shm
tmpfs                                        5.9G  1.3M  5.9G   1% /run
tmpfs                                        4.0M     0  4.0M   0% /sys/fs/cgroup
/dev/mapper/vg_root_0-lv_root_0               47G   15G   30G  33% /
tmpfs                                         15G  5.0M   15G   1% /tmp
/dev/mapper/vg_lvm_snapshot-lv_lvm_snapshot  196G   28K  186G   1% /storage/lvm_snapshot
/dev/mapper/imagebuilder_vg-imagebuilder      25G   36K   24G   1% /storage/imagebuilder
/dev/sda3                                    488M   37M  415M   9% /boot
/dev/mapper/netdump_vg-netdump               9.8G   24K  9.3G   1% /storage/netdump
/dev/mapper/core_vg-core                      49G  2.0G   45G   5% /storage/core
/dev/mapper/autodeploy_vg-autodeploy          25G   40K   24G   1% /storage/autodeploy
/dev/mapper/vtsdblog_vg-vtsdblog              25G   33M   24G   1% /storage/vtsdblog
/dev/sda2                                     10M  2.0M  8.1M  20% /boot/efi
/dev/mapper/lifecycle_vg-lifecycle            98G  3.9G   90G   5% /storage/lifecycle
/dev/mapper/vtsdb_vg-vtsdb                    49G   36M   47G   1% /storage/vtsdb
/dev/mapper/updatemgr_vg-updatemgr            98G  3.2G   90G   4% /storage/updatemgr
/dev/mapper/archive_vg-archive                98G  8.5G   85G  10% /storage/archive
/dev/mapper/log_vg-log                        25G  2.3G   21G  10% /storage/log
/dev/mapper/db_vg-db                          25G  285M   23G   2% /storage/db
/dev/mapper/dblog_vg-dblog                    25G  145M   24G   1% /storage/dblog
/dev/mapper/seat_vg-seat                      49G  154M   47G   1% /storage/seat
overlay                                       47G   15G   30G  33% /storage/containers/vc-ws1a-broker/db#######################################c0/rootfs

All services show as Running when checked via service-control --status --all

Sample output

Running:
 applmgmt lookupsvc lwsmd observability observability-vapi pschealth vc-ws1a-broker vlcm vmafdd vmcad vmdird vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-content-library vmware-eam vmware-envoy vmware-envoy-hgw vmware-envoy-sidecar vmware-hvc vmware-infraprofile vmware-perfcharts vmware-pod vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-sps vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-updatemgr vmware-vapi-endpoint vmware-vdtc vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-ui vstats vtsdb wcp
Stopped:
 vmcam vmonapi vmware-imagebuilder vmware-netdumper vmware-rbd-watchdog vmware-vcha

Certificate Check: Running the vecs-cli command reveals one or more Solution User Certificates (machine, vpxd, etc.) are in an Expired state.-Refer Verify and resolve expired vCenter Server certificates using the command line interface

for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

Sample output

[*] Store : machine
Alias : machine
Not After : <less than today's date>
[*] Store : vpxd
Alias : vpxd
Not After : : <less than today's date>
[*] Store : vpxd-extension
Alias : vpxd-extension
Not After : : <less than today's date>
[*] Store : vsphere-webclient
Alias : vsphere-webclient
Not After : : <less than today's date>

Environment

VMware vCenter Server

Cause

The issue occurs because the Solution User certificates (such as machine, vsphere-webclient, vpxd, vpxd-extension, etc.) has expired. These certificates are required for authentication and internal service communication within the vCenter appliance and when expired, the login process may hang indefinitely.

Resolution

Regenerate the expired Solution User Certificates to resolve the infinite loading screen in the vSphere Web Client.

Steps to follow:

Take a snapshot of the vCenter VM.

Identify Expired Certificates

Connect to the vCenter Server via SSH.
Type shell to enter the Bash prompt.

Run the following command to check the status of all certificate stores:

for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

Regenerate Solution User Certificates
1. If the "Not After" date has passed for your solution users, follow these steps:
2. Launch the VMware Certificate Manager utility:
```
/usr/lib/vmware-vmca/bin/certificate-manager
```
3. Select Option 6 (Replace Solution user certificates with VMCA Certificates).
4. Follow the prompts to provide the necessary information (IP, FQDN, Organization, etc.).
  
  The utility will replace the certificates and attempt to restart services.
Verify and Restart Services
- If the utility does not restart services automatically, or to ensure a clean state, run:
```
service-control --stop --all && service-control --start --all
```
Clear your browser cache and re-attempt the login to the vSphere Web Client.