SDDC Manager UI loads with error : Server failed to start. Check the SDDC Manager UI Server logs for more details.

Products

• VMware SDDC Manager

Issue/Introduction

• SDDC Manager UI loads with error : Server failed to start. Check the SDDC Manager UI Server logs for more details.
  
  
• Under SDDC Manager's /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log, below log snippets are found:
  

  YYYY-MM-DDTHH:MM:SS VERBOSE [###########] [servers/errorStateServer/index.js, error-server-init, null:44] Incoming request for error server {"url":"/ui/sddc-manager/inventory/domains/mgmt-vi-domains(monitoring-panel:monitoring/tasks)"}
YYYY-MM-DDTHH:MM:SS VERBOSE [###########] [servers/errorStateServer/index.js, error-server-init, null:44] Incoming request for error server {"url":"/ui/error"}
YYYY-MM-DDTHH:MM:SS ERROR [###########] [services/errorHandling.js, error-server-init, errorStateHandler:47] Server is in error state; serving up error html {"serverErrorState":{"code":null,"state":"error"}}
YYYY-MM-DDTHH:MM:SS VERBOSE [###########] [servers/errorStateServer/index.js, error-server-init, null:44] Incoming request for error server {"url":"/ui/"}
YYYY-MM-DDTHH:MM:SS VERBOSE [###########] [servers/errorStateServer/index.js, error-server-init, null:44] Incoming request for error server {"url":"/ui/sddc-manager/inventory/domains/mgmt-vi-domains(monitoring-panel:monitoring/tasks)"}
YYYY-MM-DDTHH:MM:SS VERBOSE [###########] [servers/errorStateServer/index.js, error-server-init, null:44] Incoming request for error server {"url":"/ui/error"}
YYYY-MM-DDTHH:MM:SS ERROR [###########] [services/errorHandling.js, error-server-init, errorStateHandler:47] Server is in error state; serving up error html {"serverErrorState":{"code":null,"state":"error"}}
YYYY-MM-DDTHH:MM:SS VERBOSE [###########] [servers/errorStateServer/index.js, error-server-init, null:44] Incoming request for error server {"url":"/ui/"}

  YYYY-MM-DDTHH:MM:SS ERROR [###########] [services/ws1bAuthentication/ws1bAuthenticationService.js, init, getWs1bClientId:95] error during getWs1bClientId Request failed with status code 500 {"body":{"arguments":["Identity Internal Server Error"],"causes":[{"message":"Failed to get the IDP type due to Identity Internal Server Error.","type":"com.vmware.evo.sddc.common.core.error.FederatedBrokerIdentityException"},{"message":"Identity Internal Server Error","type":"com.vmware.evo.sddc.identity.model.error.PscException"},{"message":"Unable to obtain Security Token Service from SSO '<vCenter fqdn>'","type":"com.vmware.evo.sddc.common.services.psc.exception.PscException"},{"message":"Server returned 'request expired' less than 0 seconds after request was issued, but it shouldn't have expired for at least 600 seconds.","type":"com.vmware.vim.sso.client.exception.TimeSynchronizationException"}],"errorCode":"SDDCMANAGER_GET_OIDC_INFO_FAILED","referenceToken":"######"},"code":"SDDCMANAGER_GET_OIDC_INFO_FAILED","stack":{"code":"ERR_BAD_RESPONSE","config":{"adapter":["xhr","http"],"env":{},"headers":{"Accept":"application/json, text/plain, */*","Accept-Encoding":"gzip, compress, deflate, br","Content-Type":"application/json","User-Agent":"axios/1.6.2","X-B3-SpanId":"###########","X-B3-TraceId":"###########"},"maxBodyLength":-1,"maxContentLength":-1,"method":"get","timeout":0,"transformRequest":[null],"transformResponse":[null],"transitional":{"clarifyTimeoutError":false,"forcedJSONParsing":true,"silentJSONParsing":true},"url":"http://127.0.0.1/v1/identity-broker/sddc-manager-oidc","xsrfCookieName":"XSRF-TOKEN","xsrfHeaderName":"X-XSRF-TOKEN"},"message":"Request failed with status code 500","name":"AxiosError","status":500},"statusCode":500}

• Under SDDC Manager's /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log, below log snippets are found:
  
  YYYY-MM-DDTHH:MM:SS INFO  [common,########,####] [c.v.e.s.i.s.PscInventoryServiceEmbeddedImpl,http-nio-127.0.0.1-####-exec-5] Returned PSCs: [Psc(domainId=########-####-####-####-############, vmName=<vCenter hostname>, vmManagementIpAddress=<vCenter IP Address>, vmHostname=<vCenter fqdn>, isReplica=false, port=443, ssoDomain=<SSO Domain>, subDomain=<SSO Domain>, sshHostKeyType=null, sshHostKey=null, bundleRepoDatastore=lcm-bundle-repo, datastoreName=<Datastore Name>, status=ACTIVE, version=<vCenter version>, deploymentType=EMBEDDED)]
YYYY-MM-DDTHH:MM:SS INFO  [common,########,####] [c.v.e.s.i.s.services.PscServiceImpl,http-nio-127.0.0.1-####-exec-5] Creating admin client using psc host <vCenter fqdn>
YYYY-MM-DDTHH:MM:SS WARN  [common,########,####] [c.v.v.s.a.c.v.impl.AbstractClient,http-nio-127.0.0.1-####-exec-5] ******* WARNING ****** WARNING ****** WARNING *******
YYYY-MM-DDTHH:MM:SS WARN  [common,########,####] [c.v.v.s.a.c.v.impl.AbstractClient,http-nio-127.0.0.1-####-exec-5] Possible remote API mismatch detected. Operation will continue, but errors are likely.
YYYY-MM-DDTHH:MM:SS WARN  [common,########,####] [c.v.v.s.a.c.v.impl.AbstractClient,http-nio-127.0.0.1-####-exec-5] ******* WARNING ****** WARNING ****** WARNING *******
YYYY-MM-DDTHH:MM:SS INFO  [common,########,####] [c.v.v.s.a.c.v.impl.AdminClientImpl,http-nio-127.0.0.1-####-exec-5] Client was created successfully
YYYY-MM-DDTHH:MM:SS INFO  [common,########,####] [c.v.v.s.a.c.v.impl.AbstractClient,http-nio-127.0.0.1-####-exec-5] Client was disposed successfully
YYYY-MM-DDTHH:MM:SS INFO  [common,########,####] [c.v.e.s.a.u.a.r.CertificateController,http-nio-127.0.0.1-####-exec-3] Inside getCertificateConfiguration
YYYY-MM-DDTHH:MM:SS ERROR [common,########,####] [c.v.v.s.client.impl.SoapBindingImpl,http-nio-127.0.0.1-####-exec-5] SOAP fault
com.sun.xml.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: The time now <Day><Month><Date>HH:MM:SS <TimeZone> <YEAR> does not fall in the request lifetime interval extended with clock tolerance of 600000 ms: [ <Day><Month><Date>HH:MM:SS <TimeZone> <YEAR>; <Day><Month><Date>HH:MM:SS <TimeZone> <YEAR>). This might be due to a clock skew problem. Please see the server log to find more
 detail regarding exact cause of the failure.
        at com.sun.xml.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:163)
YYYY-MM-DDTHH:MM:SS INFO  [common,########,####] [c.v.v.s.c.i.SecurityTokenServiceImpl$RequestResponseProcessor,http-nio-127.0.0.1-####-exec-5] Request message has expired. Server message: ns0:MessageExpired: The time now <Day><Month><Date>HH:MM:SS <TimeZone> <YEAR> does not fall in the request lifetime interval extended with clock tolerance of 600000 ms: [ <Day><Month><Date>HH:MM:SS <TimeZone> <YEAR>; <Day><Month><Date>HH:MM:SS <TimeZone> <YEAR>). This might be due to a clock skew problem.
YYYY-MM-DDTHH:MM:SS INFO  [common,########,####] [c.v.v.s.c.i.SecurityTokenServiceImpl$RequestResponseProcessor,http-nio-127.0.0.1-####-exec-5] Server returned 'request expired' less than 0 seconds after request was issued, but it shouldn't have expired for at least 600 seconds.
YYYY-MM-DDTHH:MM:SS ERROR [common,########,####] [c.v.e.s.i.s.services.PscServiceImpl,http-nio-127.0.0.1-####-exec-5] Error while creating admin client using psc <vCenter fqdn>
YYYY-MM-DDTHH:MM:SS ERROR [common,########,####] [c.v.e.s.i.s.services.PscServiceImpl,http-nio-127.0.0.1-####-exec-5] No psc host avialable.
YYYY-MM-DDTHH:MM:SS ERROR [common,########,####] [c.v.e.s.i.s.IdentityProviderServiceImpl,http-nio-127.0.0.1-####-exec-5] Failed to get the IDP type Identity Internal Server Error
YYYY-MM-DDTHH:MM:SS ERROR [common,########,####] [c.v.e.s.i.r.a.c.v.IdentityProviderController,http-nio-127.0.0.1-####-exec-5] Unable to get sddc manager oidc information due to Identity Internal Server Error
YYYY-MM-DDTHH:MM:SS ERROR [common,########,####] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-####-exec-5] [#####] SDDCMANAGER_GET_OIDC_INFO_FAILED
com.vmware.evo.sddc.identity.rest.api.error.IdentityRestServiceException:
        at com.vmware.evo.sddc.identity.rest.api.controller.v1.IdentityProviderController.getSddcWs1bOidcInfo(IdentityProviderController.java:444)
YYYY-MM-DDT04:48:32.404+0000 ERROR [common,########,####] [c.v.e.s.i.s.VcfServiceInventoryServiceImpl,http-nio-127.0.0.1-####-exec-6] Error while trying to retrieve service http://127.0.0.1/sddc-manager-ui-app/about status, 500 Internal Server Error: "<html><EOL>  <head><EOL>    <title>Server failed to start !</title><EOL> <style type=

Environment

VMware Cloud Foundation 5.x

Cause

A time synchronization (time skew) issue between the vCenter Server and the SDDC Manager node.

Resolution

• Verify Time Sync :

Check if either the vCenter Server or the SDDC Manager node is out of sync. If the SDDC Manager time is incorrect and the UI is not accessible to verify or update NTP settings, proceed with the steps below.

• Manually Set Time on SDDC Manager :

Align the SDDC Manager time with vCenter using the following command (replace <date-time> with the correct UTC timestamp based on your vCenter):

date --set="<YYYY-MM-DD HH:MM:SS UTC>"

Example: date --set="YYYY-MM-DD 04:30:00 UTC"

• Restart SDDC Manager Services

Once the time is corrected, restart the SDDC Manager services using: /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

• Validate and Update NTP Settings :

After the UI becomes accessible, log in and review the NTP configuration

• Ensure the SDDC Manager is configured with the correct NTP servers to prevent future time drift.