"Signing certificate is not valid" or "No healthy upstream" error in vCenter Server Appliance
search cancel

"Signing certificate is not valid" or "No healthy upstream" error in vCenter Server Appliance

book

Article ID: 316619

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

In an environment with a vCenter Server Appliance (VCSA) 6.5.x, 6.7.x or vCenter Server 7.0.x, 8.0.x these symptoms may appear:

  • Upon reboot of vCenter services fail to start.
  • Logging in to the vSphere Client fails with the error: HTTP Status 400 – Bad Request Message BadRequest, Signing certificate is not valid
  • In the /var/log/vmware/vpxd-svcs/vpxd-svcs.log file, there may be entries similar to:

    ERROR com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor opId=] Server rejected the provided time range. Cause:ns0:InvalidTimeRange: The token authority rejected an issue request for TimePeriod [startTime=Date MM DD:TT:SS EST YYYY, endTime=Date MM DD:TT:SS EST YYYY] :: Signing certificate is not valid at Date MM DD:TT:SS EST YYYY, cert validity: TimePeriod [startTime=Date MM DD:TT:SS EST YYYY, endTime=Date MM DD:TT:SS EST YYYY]


  • In /var/log/vmware/sso/vmware-identity-sts.log you may see errors similar to:  
    ERROR sts[##:tomcat-http--##] Throwing InvalidTimeRangeException! The token authority rejected an issue request for time period [startTime=Date MM DD:TT:SS EST YYYY, endTime=Date MM DD:TT:SS EST YYYY] :: Signing certificate is not valid
     
  • Logging in through the Web client display errors similar to: 
    503 Service Unavailable (Failed to connect to endpoint:
[N7Vmacore4Http20NamedPipeServiceSpecE:0x00007fb444041040]_serverNamespace
=/ action = Allow_pipeName =/var/run/vmware/vpxd-webserver-pipe)


  • Logging in through the Web Client displays a message similar to: "Username and password are required"
  • Replacing other certificates in the environment fails due to services not starting caused by expired sts
  • Adding, modifying or deleting registrations from the Lookup Service manually using the lsdoctor tool fails.
  • Logging in through the Web client displays errors similar to: 
    Cannot connect to vCenter Single Sign-On server https://VC_FQDN/sts/STSService/vsphere.local
OR
Cannot connect to vCenter Single Sign-On server https://VC_FQDN:7444/sts/STSService/vsphere.local
OR
[400] An error occurred while sending an authentication request to the vCenter Single Sign-On server

    Connecting services with VCSA fails with vpxd authorization errors similar to:
    [YYYY-MM-DDTHH:MM:SS] info vpxd[12853] [Originator@6876 sub=vpxCryptopID=###-########] Failed to read X509 cert; err: 151441516


  • Trying to export a VM as OVF fails, and /var/log/vmware/content-library/cls.log contains the following error:
    [YYYY-MM-DDTHH:MM:SS] [INFO ] http-nio-####-exec-#### ######## ####### ###### com.vmware.vise.security.spring.DefaultAuthenticationProvider     Session initialization complete for sessionId ######, clientId ######
[YYYY-MM-DDTHH:MM:SS] [INFO ] http-nio-####-exec-#### com.vmware.vapi.security.AuthenticationFilter                     Authentication failed com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) => {
        at com.vmware.cis.data.service.session.SessionAuthenticationHandler.authenticate(SessionAuthenticationHandler.java:36)
        at com.vmware.vapi.security.AuthenticationFilter.invoke(AuthenticationFilter.java:233)

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 9.0.x
 

Cause

  1. These issues occur when the Security Token Service (STS) certificate has expired or its signing root certificate has expired.
  2. Expired STS certificate causes internal services and solution users to not be able to acquire valid tokens and as a result fails to function as expected.
  3. Problems are not limited to certificate expiration corruption can occur in the environment, additionally sometimes multiple sts certificates are observed and renewal using vCert tool will address. 

    Note: When the STS certificate expires, it does so without warning. The expiry generally occurs two years from the initial creation or when its own signing certificate has expired which is variable depending on the certificate set up of the environment e.g vmca/custom ect.

Resolution

NOTE: Ensure to take a no memory snapshot of the vCenter server if it is in standalone mode or powered off snapshots of all vCenter servers in the same SSO domain if they are in linked mode.

The script previously attached to this KB is deprecated.

Use the new improved certificate management tool:  vCert - Scripted vCenter Expired Certificate Replacement for all certificate management/replacement workflow. 

  • Download and install vCert on the vCenter Server Appliance as described in Installation Section.
  • Checking the STS signing certificate.
  • Replacing STS signing certificate.

Additional Information

Powered by Wolken Software