"Signing certificate is not valid" or "No healthy upstream" error in vCenter Server Appliance
search cancel

"Signing certificate is not valid" or "No healthy upstream" error in vCenter Server Appliance

book

Article ID: 316619

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

A vCenter Server Appliance (VCSA) on versions 7.x, 8.x or 9.x may experience the following symptoms:

  • After a reboot of vCenter, services fail to start.
  • Logging in to the vSphere Client fails and may display any one or more of the following error messages:

Username and password are required

OR

Cannot connect to vCenter Single Sign-On server https://VC_FQDN/sts/STSService/vsphere.local

OR

Cannot connect to vCenter Single Sign-On server https://VC_FQDN:7444/sts/STSService/vsphere.local

OR

[400] An error occurred while sending an authentication request to the vCenter Single Sign-On server

OR

HTTP Status 400 – Bad Request Message BadRequest, Signing certificate is not valid

OR

503 Service Unavailable (Failed to connect to endpoint:
[N7Vmacore4Http20NamedPipeServiceSpecE:0x00007fb444041040]_serverNamespace
=/ action = Allow_pipeName =/var/run/vmware/vpxd-webserver-pipe)

  • Replacing other certificates in the environment fails due to services not starting, which is caused by an expired STS certificate.
  • Adding, modifying, or deleting registrations from the Lookup Service manually using the lsdoctor tool fails.
  • In the /var/log/vmware/vpxd-svcs/vpxd-svcs.log file, there may be entries similar to:

ERROR com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor opId=] Server rejected the provided time range. Cause:ns0:InvalidTimeRange: The token authority rejected an issue request for TimePeriod [startTime=Date MM DD:TT:SS EST YYYY, endTime=Date MM DD:TT:SS EST YYYY] :: Signing certificate is not valid at Date MM DD:TT:SS EST YYYY, cert validity: TimePeriod [startTime=Date MM DD:TT:SS EST YYYY, endTime=Date MM DD:TT:SS EST YYYY]   

  • In the /var/log/vmware/sso/vmware-identity-sts.log file, the following error may be observed:

ERROR sts[##:tomcat-http--##] Throwing InvalidTimeRangeException! The token authority rejected an issue request for time period [startTime=Date MM DD:TT:SS EST YYYY, endTime=Date MM DD:TT:SS EST YYYY] :: Signing certificate is not valid  
 

  • Connecting services with VCSA fails with vpxd authorization errors similar to:

[YYYY-MM-DDTHH:MM:SS] info vpxd[12853] [Originator@6876 sub=vpxCryptopID=###-########] Failed to read X509 cert; err: 151441516

 

  • Attempting to export a VM as an OVF template fails and the /var/log/vmware/content-library/cls.log file contains the following error:

[YYYY-MM-DDTHH:MM:SS] [INFO ] http-nio-####-exec-#### ######## ####### ###### com.vmware.vise.security.spring.DefaultAuthenticationProvider     Session initialization complete for sessionId ######, clientId ######
[YYYY-MM-DDTHH:MM:SS] [INFO ] http-nio-####-exec-#### com.vmware.vapi.security.AuthenticationFilter                     Authentication failed com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) => {
        at com.vmware.cis.data.service.session.SessionAuthenticationHandler.authenticate(SessionAuthenticationHandler.java:36)
        at com.vmware.vapi.security.AuthenticationFilter.invoke(AuthenticationFilter.java:233)

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x
VMware vCenter Server 9.x
 

Cause

These issues occur when the Security Token Service (STS) certificate has expired or its signing root certificate has expired.

An expired STS certificate causes internal services and solution users to not be able to acquire valid tokens and as a result, fail to function as expected.

Problems - not limited to certificate expiration or corruption - can occur in the environment. Additionally, multiple STS certificates may be observed and renewing them using the vCert tool will address this.

Note: When the STS certificate expires, it does so without warning. The expiry generally occurs two years from the initial creation or when its own signing certificate has expired, which is variable depending on the certificate set up of the environment (e.g. VMCA, custom, etc.)

Resolution

NOTE: Ensure to take a snapshot without memory of the vCenter server if it is standalone or powered off snapshots of all vCenter servers in the same SSO domain if they are in linked mode.

The script previously attached to this KB is deprecated.

Use the new and improved certificate management tool:  vCert - Scripted vCenter Expired Certificate Replacement for all certificate management/replacement workflow. 

  • Download and install vCert on the vCenter Server Appliance as described in Installation Section.
  • Checking the STS signing certificate.
  • Replacing the STS signing certificate.

Additional Information

Powered by Wolken Software