"Signing certificate is not valid" or "No healthy upstream" error in vCenter Server Appliance
search cancel

"Signing certificate is not valid" or "No healthy upstream" error in vCenter Server Appliance

book

Article ID: 316619

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

vCenter Server Appliance (VCSA) versions 7.x, 8.x, or 9.x may exhibit the following symptoms:

  • Service Failures:
    • After a reboot, vCenter Server services fail to start.
    • An expired STS certificate prevents the vmware-stsd service from starting.
    • vCenter Server services fail to start with vpxd authorization errors similar to:

[YYYY-MM-DDTHH:MM:SS] info vpxd[12853] [Originator@6876 sub=vpxCryptopID=###-########] Failed to read X509 cert; err: 151441516

  • Login Errors:
    • Logging in to the vSphere Client fails with one or more of the following errors:
      • Username and password are required
      • Cannot connect to vCenter Single Sign-On server https://VC_FQDN/sts/STSService/vsphere.local
      • Cannot connect to vCenter Single Sign-On server https://VC_FQDN:7444/sts/STSService/vsphere.local
      • [400] An error occurred while sending an authentication request to the vCenter Single Sign-On server
      • HTTP Status 400 – Bad Request Message BadRequest, Signing certificate is not valid
      • 503 Service Unavailable (Failed to connect to endpoint:
        [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007fb444041040]_serverNamespace
        =/ action = Allow_pipeName =/var/run/vmware/vpxd-webserver-pipe)
      • No Healthy Upstream
  • Management Failures:
    • Adding, modifying, or deleting registrations from the Lookup Service manually using the lsdoctor tool, fails.
    • In an Enhanced Linked Mode (ELM) environment, all vCenter Server instances become inaccessible.
    • An expired STS certificates prevents vCenter services from starting, causing certificate replacements to fail.
    • Attempting to export a VM as an OVF template fails.
      • The /var/log/vmware/content-library/cls.log file of the vCenter contains the following entry:

[YYYY-MM-DDTHH:MM:SS] [INFO ] http-nio-####-exec-#### ######## ####### ###### com.vmware.vise.security.spring.DefaultAuthenticationProvider     Session initialization complete for sessionId ######, clientId ######
[YYYY-MM-DDTHH:MM:SS] [INFO ] http-nio-####-exec-#### com.vmware.vapi.security.AuthenticationFilter                     Authentication failed com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) => {
        at com.vmware.cis.data.service.session.SessionAuthenticationHandler.authenticate(SessionAuthenticationHandler.java:36)
        at com.vmware.vapi.security.AuthenticationFilter.invoke(AuthenticationFilter.java:233)

  • Other symptoms:
    • In the /var/log/vmware/vpxd-svcs/vpxd-svcs.log file of the vCenter, the following error may be observed:

ERROR com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor opId=] Server rejected the provided time range. Cause:ns0:InvalidTimeRange: The token authority rejected an issue request for TimePeriod [startTime=Date MM DD:TT:SS EST YYYY, endTime=Date MM DD:TT:SS EST YYYY] :: Signing certificate is not valid at Date MM DD:TT:SS EST YYYY, cert validity: TimePeriod [startTime=Date MM DD:TT:SS EST YYYY, endTime=Date MM DD:TT:SS EST YYYY]

    • In the /var/log/vmware/sso/vmware-identity-sts.log file, the following error may be observed:

ERROR sts[##:tomcat-http--##] Throwing InvalidTimeRangeException! The token authority rejected an issue request for time period [startTime=Date MM DD:TT:SS EST YYYY, endTime=Date MM DD:TT:SS EST YYYY] :: Signing certificate is not valid

    • The VDT (VCF Diagnostic Tool for vSphere) may indicate that there is an issue with the STS certificate (not all of the symptoms mentioned above will be observed or experienced, but it is important to note that this resolution also applies):

VC STS Certificate Check
            [FAIL]    STS Certificate Check
                        1x expired STS certificates.
                        Documentation:     https://knowledge.broadcom.com/external/article?legacyId=76719

Environment

vCenter 7.x, 8.x, 9.x
 
 

Cause

These issues occur when the Security Token Service (STS) certificate has expired or its signing root certificate has expired.

An expired STS certificate causes internal services and solution users to not be able to acquire valid tokens and as a result, fail to function as expected.

Problems - not limited to certificate expiration or corruption - can occur in the environment. Additionally, multiple STS certificates may be observed and renewing them using the vCert tool will address this.

Note: The STS certificate generally expires two years from the initial creation or when its own signing certificate has expired, which is variable depending on the certificate set up of the environment (e.g. VMCA, custom, etc.).

Resolution

NOTE: Ensure to take a snapshot without memory of the vCenter server if it is standalone or powered off snapshots of all vCenter servers in the same SSO domain if they are in linked mode.

Use the new and improved certificate management tool - vCert - Scripted vCenter Expired Certificate Replacement - for all certificate management/replacement workflows. 

  • Download and install vCert on the vCenter Server Appliance as described in the Installation Section.
  • To check the STS signing certificate:
  • To replace the STS signing certificate:

Additional Information

Powered by Wolken Software