Error: SDDC Manager UI displays blank page or Unauthorized access

Products

VMware Cloud Foundation

Issue/Introduction

Administrators encounter a blank page or a redirection to the vCenter login when accessing the SDDC Manager UI.

In the SDDC Manager log file /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log, the following error is observed::
YYYY-MM-DDTHH:MM:SS ERROR [4736320faafb4eee] [services/errorHandling.js, ####-####-########, productionErrorRoute:131]
600.158: VError: Sending error response: SAML assertion not yet valid
at Object.errorHandlerSend (/opt/vmware/vcf/####-#######-##-###/server/src/errors/VCFError.js:104:5)
at productionErrorRoute (/opt/vmware/vcf/####-#######-##-###/server/src/services/errorHandling.js:118:34)
at Layer.handle_error (/opt/vmware/vcf/####-#######-##-###/server/node_modules/express/lib/router/layer.js:71:5)
at trim_prefix (/opt/vmware/vcf/####-#######-##-###/server/node_modules/express/lib/router/index.js:326:13)
at /opt/vmware/vcf/####-#######-##-###/server/node_modules/express/lib/router/index.js:286:9
at Function.process_params (/opt/vmware/vcf/####-#######-##-###/server/node_modules/express/lib/router/index.js:346:12)
at next (/opt/vmware/vcf/####-#######-##-###/server/node_modules/express/lib/router/index.js:280:10)
at Layer.handle_error (/opt/vmware/vcf/####-#######-##-###/server/node_modules/express/lib/router/layer.js:67:12)
at trim_prefix (/opt/vmware/vcf/####-#######-##-###/server/node_modules/express/lib/router/index.js:326:13)
at /opt/vmware/vcf/####-#######-##-###/server/node_modules/express/lib/router/index.js:286:9
Error Info: {"id":"########-####-####-####-############","requestedPath":"/ui/api/internal/login/callback","error":{"id":"########-####-####-####-############"},"stack":"Error: SAML assertion not yet valid\n at SAML.checkTimestampsValidityError (/opt/vmware/vcf/####-#######-##-###/server/node_modules/########-####/lib/####-####/saml.js:932:24)\n at SAML.processValidlySignedAssertionAsync (/opt/vmware/vcf/####-#######-##-###/server/node_modules/########-####/lib/####-####/saml.js:870:33)\n at processTicksAndRejections (node:internal/process/task_queues:105:5)\n at runNextTicks (node:internal/process/task_queues:69:3)\n at process.processImmediate (node:internal/timers:453:9)\n at process.callbackTrampoline (node:internal/async_hooks:130:17)\n at async SAML.validatePostResponseAsync (/opt/vmware/vcf/####-#######-##-###/server/node_modules/########-####/lib/####-####/saml.js:587:24)","message":"SAML assertion not yet valid","status":500,"errorModule":600,"errorCode":158}
caused by:
Error: SAML assertion not yet valid
at SAML.checkTimestampsValidityError (/opt/vmware/vcf/####-#######-##-###/server/node_modules/########-####/lib/####-####/saml.js:932:24)
at SAML.processValidlySignedAssertionAsync (/opt/vmware/vcf/####-#######-##-###/server/node_modules/########-####/lib/####-####/saml.js:870:33)
at processTicksAndRejections (node:internal/process/task_queues:105:5)
at runNextTicks (node:internal/process/task_queues:69:3)
at process.processImmediate (node:internal/timers:453:9)
at process.callbackTrampoline (node:internal/async_hooks:130:17)
at async SAML.validatePostResponseAsync (/opt/vmware/vcf/####-#######-##-###/server/node_modules/########-####/lib/####-####/saml.js:587:24)
All Errors Info:
SAML assertion not yet valid {"error":{"id":"########-####-####-####-############"},"id":"########-####-####-####-############","requestedPath":"/ui/api/internal/login/callback","stack":"Error: SAML assertion not yet valid\n at SAML.checkTimestampsValidityError (/opt/vmware/vcf/####-#######-##-###/server/node_modules/########-####/lib/####-####/saml.js:932:24)\n at SAML.processValidlySignedAssertionAsync (/opt/vmware/vcf/####-#######-##-###/server/node_modules/########-####/lib/####-####/saml.js:870:33)\n at processTicksAndRejections (node:internal/process/task_queues:105:5)\n at runNextTicks (node:internal/process/task_queues:69:3)\n at process.processImmediate (node:internal/timers:453:9)\n at process.callbackTrampoline (node:internal/async_hooks:130:17)\n at async SAML.validatePostResponseAsync (/opt/vmware/vcf/####-#######-##-###/server/node_modules/########-####/lib/####-####/saml.js:587:24)","status":500}
Further investigation shows a time synchronization mismatch between vCenter Server and SDDC Manager.
vCenter NTP
SDDC Manager NTP

Environment

VMware Cloud Foundation (VCF) 4.x, 5.x, 9.x
Management vCenter Server

Cause

Clock Skew: A time difference (>60 seconds) between SDDC Manager and vCenter Server. vCenter issues a SAML token with a "NotBefore" timestamp that SDDC Manager rejects if its clock is behind.
Credential Drift: The vCenter root password was changed directly on the vCenter appliance instead of through SDDC Manager, causing authentication failures for internal API calls.

Resolution

Step 1: Verify Time Sync Discrepancy

Log in via SSH to the SDDC Manager as root.
Log in via SSH to the Management vCenter Server (vCSA) as root.
Run the date command on both appliances simultaneously:
```
date -u
```
Compare the output:
- If the SDDC Manager time is earlier than the vCenter Server time, SAML validation may fail.

Step 2: Synchronize NTP Services

Ensure both the SDDC Manager and vCenter Server are configured to use the same NTP server(s).
On the SDDC Manager, if a noticeable time drift exists, force a time synchronization:
```
systemctl stop ntpd
ntpdate -u <NTP_Server_IP>
systemctl start ntpd 
```
Verify NTP synchronization status:
```
ntpq -pn
```

Note:
If both systems are already configured to use the same NTP source, it is recommended to restart the ntpd service first.
If synchronization still fails, consider temporarily changing the NTP server address on the affected appliance to complete the sync.

Step 3: Clear Browser Cache and Retry

Clear browser cookies and cache.
Open a new Incognito / Private browser window.
Attempt to log in to the SDDC Manager UI again.

Step 4: Realign vCenter Credentials
If time is synchronized but the error persists:
1. Verify if the vCenter root password matches the SDDC Manager security inventory.
2. Use lookup_passwords to retrieve the expected password and reset the vCenter root password to match if drift is confirmed.

Step 5: Restart Services
Restart SDDC Manager orchestration services:
bash
/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

Fixed in VCF 9.0 and higher. See Download Broadcom products and software for steps to download this release.

Additional Information

Managing Passwords in VMware Cloud Foundation
Contact Broadcom Support