"Signing certificate is not valid" or "No healthy upstream" error in vCenter Server Appliance
search cancel

"Signing certificate is not valid" or "No healthy upstream" error in vCenter Server Appliance

book

Article ID: 316619

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

vCenter Server Appliance (VCSA) versions 7.x, 8.x, or 9.x may exhibit the following symptoms:

  • Service Failures:
    • After a reboot, vCenter Server services fail to start.
    • An expired STS certificate prevents the vmware-stsd service from starting.
    • vCenter Server services fail to start with vpxd authorization errors in /var/log/vmware/vpxd/vpxd.log:
      [YYYY-MM-DDTHH:MM:SS] info vpxd[12853] [Originator@6876 sub=vpxCryptopID=###-########] Failed to read X509 cert; err: #########
  • Login Errors:
    • Logging in to the vSphere Client fails with one or more of the following errors:
      • Username and password are required
      • Cannot connect to vCenter Single Sign-On server https://VC_FQDN/sts/STSService/vsphere.local
      • Cannot connect to vCenter Single Sign-On server https://VC_FQDN:7444/sts/STSService/vsphere.local
      • [400] An error occurred while sending an authentication request to the vCenter Single Sign-On server
      • HTTP Status 400 – Bad Request Message BadRequest, Signing certificate is not valid
      • 503 Service Unavailable (Failed to connect to endpoint:
        [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007fb444041040]_serverNamespace=/ action = Allow_pipeName =/var/run/vmware/vpxd-webserver-pipe)
      • No Healthy Upstream
  • Management Failures:
    • Adding, modifying, or deleting registrations from the Lookup Service manually using the lsdoctor tool, fails.
    • In an Enhanced Linked Mode (ELM) environment, all vCenter Server instances become inaccessible.
    • An expired STS certificates prevents vCenter services from starting, causing certificate replacements to fail.
    • Attempting to export a virtual machine as an OVF template fails.
      • The /var/log/vmware/content-library/cls.log file of the vCenter contains the following entry:

[YYYY-MM-DDTHH:MM:SS] [INFO ] http-nio-####-exec-#### ######## ####### ###### com.vmware.vise.security.spring.DefaultAuthenticationProvider     Session initialization complete for sessionId ######, clientId ######
[YYYY-MM-DDTHH:MM:SS] [INFO ] http-nio-####-exec-#### com.vmware.vapi.security.AuthenticationFilter                     Authentication failed com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) => {
        at com.vmware.cis.data.service.session.SessionAuthenticationHandler.authenticate(SessionAuthenticationHandler.java:36)
        at com.vmware.vapi.security.AuthenticationFilter.invoke(AuthenticationFilter.java:233)

  • Other symptoms:
    • The vmware-stsd service is stopped.

    • In the /var/log/vmware/vpxd-svcs/vpxd-svcs.log file of the vCenter, the following error may be observed:
      ERROR com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor opId=] Server rejected the provided time range. Cause:ns0:InvalidTimeRange: The token authority rejected an issue request for TimePeriod [startTime=Date MM DD:TT:SS EST YYYY, endTime=Date MM DD:TT:SS EST YYYY] :: Signing certificate is not valid at Date MM DD:TT:SS EST YYYY, cert validity: TimePeriod [startTime=Date MM DD:TT:SS EST YYYY, endTime=Date MM DD:TT:SS EST YYYY]
    • In the /var/log/vmware/sso/vmware-identity-sts.log file, the following error may be observed:
      ERROR sts[##:tomcat-http--##] Throwing InvalidTimeRangeException! The token authority rejected an issue request for time period [startTime=Date MM DD:TT:SS EST YYYY, endTime=Date MM DD:TT:SS EST YYYY] :: Signing certificate is not valid
    • The VDT (VCF Diagnostic Tool for vSphere) may indicate that there is an issue with the STS certificate (Not all of the symptoms mentioned above will be observed, but it is important to note that this resolution still applies):
      VC STS Certificate Check

            [FAIL]    STS Certificate Check
                        1x expired STS certificates.
                        Documentation:     https://knowledge.broadcom.com/external/article?legacyId=76719

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x
VMware vCenter 9.x

Cause

These issues occur when the Security Token Service (STS) certificate has expired or its signing root certificate has expired.

An expired STS certificate causes internal services and solution users to not be able to acquire valid tokens and as a result, fail to function as expected.

Environmental issues can occur beyond just certificate expiration or corruption. If multiple STS certificates are present, renewing them with the vCert tool will resolve the problem.

Note: The STS certificate generally expires two years from the initial creation or when its own signing certificate has expired, which is variable depending on the certificate set up of the environment (e.g. VMCA, custom, etc.).

Resolution

Note: A snapshot without memory must be taken if the vCenter Server is standalone, or powered-off snapshots must be taken for all vCenter Servers in the same SSO domain if operating in Enhanced Linked Mode (ELM).

Use the certificate management tool vCert - Scripted vCenter Expired Certificate Replacement for all certificate management/replacement workflows. 

  1. Download and install vCert on the vCenter Server Appliance as described in the Installation Section.
  2. To check the STS signing certificate:
      Use Option 8 -  View STS signing certificates from the menu 2: View Certificate Info.
  3. To replace the STS signing certificate:
      Use the Option 8 - STS signing certificates from the menu 3: Manage Certificates.

Additional Information

Powered by Wolken Software