"Signing certificate is not valid" error in vCenter Server Appliance

Products

VMware vCenter Server

Issue/Introduction

This article provide steps on regenerating and replacing expired Security Token Service (STS) certificate in VCSA 6.5.x, 6.7.x , 7.0.x and vCenter Server 8.0.x using a shell script.

For steps on regenerating and replacing STS certificate in VMware vCenter Server 6.5.x and 6.7.x installed on Windows using a PowerShell script, see "Signing certificate is not valid" error in vCenter Server 6.5.x and 6.7.x on Windows.

For more information on STS certificates, see Security Token Service STS

Symptoms:
In an environment with a vCenter Server Appliance (VCSA) 6.5.x, 6.7.x or vCenter Server 7.0.x, 8.0.x you experience these symptoms:

The vmware-vpxd service fails to start.

Logging in to the vSphere Client fails with the error:

HTTP Status 400 – Bad Request Message BadRequest, Signing certificate is not valid

In the /var/log/vmware/vpxd-svcs/vpxd-svcs.log file, you see entries similar to:

ERROR com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor opId=] Server rejected the provided time range. Cause:ns0:InvalidTimeRange: The token authority rejected an issue request for TimePeriod [startTime=Date MM DD:TT:SS EST YYYY, endTime=Date MM DD:TT:SS EST YYYY] :: Signing certificate is not valid at Date MM DD:TT:SS EST YYYY, cert validity: TimePeriod [startTime=Date MM DD:TT:SS EST YYYY, endTime=Date MM DD:TT:SS EST YYYY]

Note: The endTime should be a date in the past if the certificate is expired.

Logging in through the Web client display errors similar to:
Logging in through the Web Client displays a message similar to:
Replacing any certificate on either PSC or VCSA fails.
Adding, modifying or deleting registrations from the Lookup Service manually using the lsdoctor tool fails.
Deploying a new PSC and doing a cross-domain repoint fails.
Deploying a new PSC as a replication partner on the existing SSO domain fails.

Logging in through the Web client displays errors similar to:

Cannot connect to vCenter Single Sign-On server https://VC_FQDN/sts/STSService/vsphere.local

OR

Cannot connect to vCenter Single Sign-On server https://VC_FQDN:7444/sts/STSService/vsphere.local

OR

[400] An error occurred while sending an authentication request to the vCenter Single Sign-On server

Connecting services with VCSA fails with vpxd authorization errors similar to:

[YYYY-MM-DDTHH:MM:SS] info vpxd[12853] [Originator@6876 sub=vpxCryptopID=SWI-7203af8f] Failed to read X509 cert; err: 151441516

trying to export a VM as OVF fails, and /var/log/vmware/content-library/cls.log contains the following error:

cls.log 
[YYYY-MM-DDTHH:MM:SS] [INFO ] http-nio-5090-exec-1022      70179544 103561 ###### com.vmware.vise.security.spring.DefaultAuthenticationProvider     Session initialization complete for sessionId 103561, clientId 200264 
[YYYY-MM-DDTHH:MM:SS] [INFO ] http-nio-5090-exec-1022       com.vmware.vapi.security.AuthenticationFilter                     Authentication failed com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) => {
        at com.vmware.cis.data.service.session.SessionAuthenticationHandler.authenticate(SessionAuthenticationHandler.java:36)
        at com.vmware.vapi.security.AuthenticationFilter.invoke(AuthenticationFilter.java:233)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 6.5.x

VMware vCenter Server 6.7.x

VMware vCenter Server 7.0.x

VMware vCenter Server 8.0.x

Cause

These issue occurs when the Security Token Service (STS) certificate has expired. This causes internal services and solution users to not be able to acquire valid tokens and as a result fails to function as expected.

Note: When the STS certificate expires, it does so without warning. On some systems, this expiry may occur as soon as two years from the initial deployment.

The following scenarios can cause STS signing certificate to expire at 2 years:

Fresh installation of PSC/vCenter Server 6.5 starting with U2 or later (6.5 only).
Fresh installation of PSC/vCenter Server 6.5 U2 or any later 6.5 releases and upgraded to a later version including 6.7 and 7.0.
STS signing certificate has been replaced using certool post-installation of PSC or vCenter Server.
STS signing certificate has been replaced with custom certificate (Internal/External CA Signed).
STS certificate generated with the fixsts.sh script.

Resolution

To resolve the Signing certificate is not valid error:

Download the attached fixsts.sh script from this article and upload to the impacted PSC or vCenter Server with Embedded PSC to the /tmp folder.
If the connection to upload to the vCenter by the SCP client is rejected, run this from an SSH session to the vCenter:
```
# chsh -s /bin/bash
```
Connect to the PSC or vCenter Server with an SSH session if you have not already per Step 2.
Navigate to the /tmp directory:
```
# cd /tmp
```
make the file executable:
```
# chmod +x fixsts.sh
```
Run the script:
```
# ./fixsts.sh
```
Restart services on all vCenters and/or PSCs in your SSO domain by using below commands:
```
# service-control --stop --all && service-control --start --all
```
Note: Restart of services will fail if there are other expired certificates like Machine SSL or Solution User. Proceed with the next step to identify and replace expired certificates.

Check for expiration and replace any other expired certificates you might have, using certificate manager as shown in How to use vSphere Certificate Manager to Replace SSL Certificates or follow Option 8 as shown in How to regenerate vSphere 6.x certificates using self-signed VMCA if both Machine SSL and Solution User certificates are expired.

Notes:

The following one-liner can determine other expired certificates for the vCenter Server Appliance:

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done

If Machine SSL or VMCA Root certificates has been replaced, then it will need to re-register 2nd party solutions such as NSX, SRM, and vSphere Replication.

If HLM (Hybrid Linked Mode) is in use without a gateway, it would need to re-sync the certs from Cloud to On-Prem after following this procedure.

The script will ask for the SSO administrator password and then proceed to regenerate and replace STS certificate.

For example:

NOTE: This works on external and embedded PSCs
This script will do the following
1: Regenerate STS certificate
What is needed?
1: Offline snapshots of VCs/PSCs
2: SSO Admin Password
IMPORTANT: This script should only be run on a single PSC per SSO domain
==================================
Resetting STS certificate for <vc_fqdn> started on Fri May 22 14:39:40 UTC 2020

Detected DN: cn= <vc_fqdn>,ou=Domain Controllers,dc=vsphere,dc=local
Detected PNID: <vc_fqdn>
Detected PSC: <vc_fqdn>
Detected SSO domain name: vsphere.local
Detected Machine ID: ce510c87-35e6-444e-82f0-60a7527608a3
Detected IP Address: <vc IP Address>
Domain CN: dc=vsphere,dc=local
==================================
==================================

Detected Root's certificate expiration date: 2030 May 16
Detected today's date: 2020 May 22
==================================

Exporting and generating STS certificate

Status : Success
Using config file : /tmp/vmware-fixsts/certool.cfg
Status : Success

Enter password for [email protected]:
Amount of tenant credentials: 1
Exporting tenant and trustedcertchain 1 to /tmp/vmware-fixsts

Deleting tenant and trustedcertchain 1

Applying newly generated STS certificate to SSO domain
adding new entry "cn=TenantCredential-1,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local"

adding new entry "cn=TrustedCertChain-1,cn=TrustedCertificateChains,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local"

Replacement finished - Please restart services on all vCenters and PSCs in your SSO domain
==================================
IMPORTANT: In case you're using HLM (Hybrid Linked Mode) without a gateway, you would need to re-sync the certs from Cloud to On-Prem after following this procedure
==================================
==================================

Note: If receive the following error when trying to run the script:

bash: ./fixsts.sh: /bin/bash^M: bad interpreter: No such file or directory

This error is caused by DOS carriage returns added to the script when copying from a Windows-based text editor. To resolve this problem, run this command and rerun the script:

# sed -i -e 's/\r$//' fixsts.sh

Notes:

If upgraded from vSphere 5.x or 6.0, there may have been multiple STS chains (trustedcertchain) present in TenantCredentials. When the nodes which issued the other chains are no longer accessible, this can cause authorization errors.
Running the script will delete all the old STS chains and replace with a single new chain, resolving the authorization errors caused by the obsolete chains.
If there are multiple certificate chains and no LEAF in the (trustedcertchain) please proceed to regenerate the STS Certificate.

Additional Information

For more information on viewing the STS certificate and determining the expiry date, see Checking Expiration of STS Certificate on vCenter Server.
VMware Skyline Health Diagnostics for vSphere - FAQ

Impact/Risks:

Warning

This script interacts with the VMDIR's database. Take an offline snapshot concurrently for all vCenter Servers and Platform Service Controllers in the SSO domain before running the script. Failing to do so may result in an unrecoverable error and require redeploying vCenter Server.

Notes:

This script should only be run once per SSO domain.
In environments containing Horizon View, see Connection Server unable to accept vCenter thumbprint with an error "There was an error identifying the validity of the server" (67701).

Attachments

fixsts get_app