VCF Operations Fleet UI Blank and API Failures Following vSAN Graceful Shutdown

Products

• VCF Operations

Issue/Introduction

Following a maintenance window involving a vSAN graceful cluster shutdown and out-of-band password changes, a VMware Cloud Foundation (VCF) environment experiences cascading API and UI failures:

1. The Management Domain is missing from the Licensing Management -> Licenses view in VCF Operations.

2. Both the Fleet Management -> Certificates and Fleet Management -> Passwords views are blank in VCF Operations.

3. When viewing Workload Domains in SDDC Manager, an error banner displays: "Unable to fetch license".

4. The SDDC Manager UI occasionally throws 503 Service Unavailable or 502 Bad Gateway errors.

5. In the operationsmanager.log, errors regarding JWT expiration are present:

Unknown exception while parsing jwt token JWT expired ........ Difference of 1 millisecond allowed clock skew: 0 milliseconds.

6. In the vcf-commonsvcs.log, gateway errors appear:

Error while trying to retrieve service [http://127.0.0.1/operationsmanager/about](http://127.0.0.1/operationsmanager/about) status, 502 Bad Gateway
Error while trying to retrieve service [http://127.0.0.1/sddc-manager-ui-app/about](http://127.0.0.1/sddc-manager-ui-app/about) status, 503 Service Unavailable

Environment

VCF Operations 9.0x

Cause

This issue is caused by a chain of events starting with a known vSAN advanced setting issue and exacerbated by subsequent administrative actions:

1. vSAN Pause State: During a graceful vSAN cluster shutdown, the shutdown script sets the advanced host attribute DOMPauseAllCCPs to 1 (pausing cluster functions). Upon restart, the script fails to revert this to 0. The ESXi hosts boot, but the vSAN datastore remains completely paused and inaccessible to VMs.

2. NTP Time Drift and JWT Expiration: Because the management VMs (vCenter, SDDC Manager, VCF Operations) are suspended on an inaccessible vSAN datastore, their internal clocks experience severe drift. VMware's internal APIs rely on time-sensitive JSON Web Tokens (JWT). The time drift instantly invalidates these tokens, causing the inventory service to fail and the SDDC Manager UI to throw 502/503 errors.

3. Authentication Lockout: During the maintenance window, ESXi root passwords were changed directly via vCenter Host Profiles (out-of-band). SDDC Manager must be the source of truth for passwords. This out-of-band change severed SDDC Manager's ability to authenticate with the hosts to pull inventory, triggering the "Unable to fetch license" banner.

Resolution

Step 1: Release the vSAN Pause State (see kb: 315520)

SSH into each ESXi host in the affected vSAN cluster.

a. Run the following command to check if the vSAN components are paused: esxcfg-advcfg -g /VSAN/DOMPauseAllCCPs

b. If the value is 1, revert it to 0 by running: esxcfg-advcfg -s 0 /VSAN/DOMPauseAllCCPs

c. Confirm VMs are now accessible and I/O has resumed.

 

Step 2: Correct NTP Time Drift
Time synchronization must be strictly enforced across all management components to allow JWT tokens to validate.

a. SSH into the SDDC Manager appliance, VCF Operations appliance, and vCenter Server.

b. Verify the current time using the date command.

c. Force an NTP sync with your designated NTP server (e.g., using ntpq -p or the VAMI interface).

d. Once time is verified to be identical across components, perform a rolling reboot of the management appliances, starting with vCenter, followed by SDDC Manager and VCF Operations, to generate fresh API tokens.

 

Step 3: Remediate SDDC Manager Passwords
With the APIs communicating, SDDC Manager's database must be synced with the new out-of-band passwords.

Note: Attempting to update passwords via API before correcting the NTP time drift will fail due to JWT invalidation.

a.Log into the restored SDDC Manager UI.

b. Navigate to Administration > Security > Password Management (or the corresponding Fleet Passwords GUI).

c. Select the ESXi hosts whose passwords were changed via Host Profiles.

d. Use the Remediate function and provide the new root password. This updates the SDDC Manager database and restores host inventory sync, clearing the "Unable to fetch license" banner.

Additional Information

Related Information
VMware KB 418912: SDDC Manager UI Displays "Unable to fetch license" due to disconnected accounts or locked out credentials.

VMware vSAN Documentation: Handling inconsistent cluster power status in vSAN shutdown workflows.

VMware Cloud Foundation Administration Guide: Managing and rotating passwords using SDDC Manager.