"Signing certificate is not valid" error in vCenter Server Appliance

Products

VMware vCenter Server

Issue/Introduction

This article provides steps on regenerating and replacing expired Security Token Service (STS) certificate in VCSA 6.5.x, 6.7.x , 7.0.x and vCenter Server 8.0.x using a shell script.

For steps on regenerating and replacing STS certificate in VMware vCenter Server 6.5.x and 6.7.x installed on Windows using a PowerShell script, see "Signing certificate is not valid" error in vCenter Server 6.5.x and 6.7.x on Windows.

For more information on STS certificates, see Security Token Service STS

In an environment with a vCenter Server Appliance (VCSA) 6.5.x, 6.7.x or vCenter Server 7.0.x, 8.0.x these symptoms may appear:

The vmware-vpxd service fails to start.
Logging in to the vSphere Client fails with the error: HTTP Status 400 – Bad Request Message BadRequest, Signing certificate is not valid
In the /var/log/vmware/vpxd-svcs/vpxd-svcs.log file, there may be entries similar to:

ERROR com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor opId=] Server rejected the provided time range. Cause:ns0:InvalidTimeRange: The token authority rejected an issue request for TimePeriod [startTime=Date MM DD:TT:SS EST YYYY, endTime=Date MM DD:TT:SS EST YYYY] :: Signing certificate is not valid at Date MM DD:TT:SS EST YYYY, cert validity: TimePeriod [startTime=Date MM DD:TT:SS EST YYYY, endTime=Date MM DD:TT:SS EST YYYY]
Logging in through the Web client display errors similar to:
Logging in through the Web Client displays a message similar to:
Replacing any certificate on either PSC or VCSA fails.
Adding, modifying or deleting registrations from the Lookup Service manually using the lsdoctor tool fails.
Deploying a new PSC and doing a cross-domain repoint fails.
Deploying a new PSC as a replication partner on the existing SSO domain fails.
Logging in through the Web client displays errors similar to:

Cannot connect to vCenter Single Sign-On server https://VC_FQDN/sts/STSService/vsphere.local

OR

Cannot connect to vCenter Single Sign-On server https://VC_FQDN:7444/sts/STSService/vsphere.local

OR

[400] An error occurred while sending an authentication request to the vCenter Single Sign-On server

Connecting services with VCSA fails with vpxd authorization errors similar to:

[YYYY-MM-DDTHH:MM:SS] info vpxd[12853] [Originator@6876 sub=vpxCryptopID=SWI-7203af8f] Failed to read X509 cert; err: 151441516
trying to export a VM as OVF fails, and /var/log/vmware/content-library/cls.log contains the following error:

[YYYY-MM-DDTHH:MM:SS] [INFO ] http-nio-5090-exec-1022 70179544 103561 ###### com.vmware.vise.security.spring.DefaultAuthenticationProvider Session initialization complete for sessionId 103561, clientId 200264
[YYYY-MM-DDTHH:MM:SS] [INFO ] http-nio-5090-exec-1022 com.vmware.vapi.security.AuthenticationFilter Authentication failed com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) => {
at com.vmware.cis.data.service.session.SessionAuthenticationHandler.authenticate(SessionAuthenticationHandler.java:36)
at com.vmware.vapi.security.AuthenticationFilter.invoke(AuthenticationFilter.java:233)

Environment

VMware vCenter Server 6.5.x

VMware vCenter Server 6.7.x

VMware vCenter Server 7.0.x

VMware vCenter Server 8.0.x

Cause

These issues occur when the Security Token Service (STS) certificate has expired. This causes internal services and solution users to not be able to acquire valid tokens and as a result fails to function as expected.

Note: When the STS certificate expires, it does so without warning. On some systems, this expiry may occur as soon as two years from the initial deployment.

The following scenarios can cause STS signing certificate to expire at 2 years:

Fresh installation of PSC/vCenter Server 6.5 starting with U2 or later (6.5 only).
Fresh installation of PSC/vCenter Server 6.5 U2 or any later 6.5 releases and upgraded to a later version including 6.7 and 7.0.
STS signing certificate has been replaced using certool post-installation of PSC or vCenter Server.
STS signing certificate has been replaced with custom certificate (Internal/External CA Signed).
STS certificate generated with the fixsts.sh script.

Resolution

To resolve the Signing certificate is not valid error:

Connect to the PSC or vCenter Server with an SSH session with the root user.
Download the attached fixsts.sh script from this article and upload to the impacted PSC or vCenter Server with Embedded PSC to the /tmp folder.
If the connection to upload to the vCenter by the SCP client is rejected, run this from an SSH session to the vCenter:
```
# chsh -s /bin/bash root
```
Navigate to the /tmp directory:
```
# cd /tmp
```
make the file executable:
```
# chmod +x fixsts.sh
```
Run the script:
```
# ./fixsts.sh
```
Restart services on all vCenters and/or PSCs in your SSO domain by using below commands:
```
# service-control --stop --all && service-control --start --all
```
Note: Restart of services will fail if there are other expired certificates like Machine SSL or Solution User. Proceed with the next step to identify and replace expired certificates.
Check for expiration and replace any other expired certificates, using certificate manager as shown in How to use vSphere Certificate Manager to Replace SSL Certificates or follow Option 8 as shown in How to regenerate vSphere 6.x certificates using self-signed VMCA if both Machine SSL and Solution User certificates are expired.

Notes:

The following one-liner can determine other expired certificates for the vCenter Server Appliance:

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done

If Machine SSL or VMCA Root certificates has been replaced, then it will need to re-register 2nd party solutions such as NSX, SRM, and vSphere Replication.

If HLM (Hybrid Linked Mode) is in use without a gateway, it would need to re-sync the certs from Cloud to On-Prem after following this procedure.
The script will ask for the SSO administrator password and then proceed to regenerate and replace STS certificate.

For example

NOTE: This works on external and embedded PSCs

This script will do the following

1: Regenerate STS certificate

What is needed?

1: Offline snapshots of VCs/PSCs

2: SSO Admin Password

IMPORTANT: This script should only be run on a single PSC per SSO domain

==================================

Resetting STS certificate for <vc_fqdn> started on [YYYY-MM-DDTHH:MM:SS]

Detected DN: cn= <vc_fqdn>,ou=Domain Controllers,dc=vsphere,dc=local

Detected PNID: <vc_fqdn>

Detected PSC: <vc_fqdn>

Detected SSO domain name: vsphere.local

Detected Machine ID: <machineID>

Detected IP Address: <vc IP Address>

Domain CN: dc=vsphere,dc=local

==================================

==================================Detected Root's certificate expiration date: 2030 May 16

Detected today's date: YYYY MMM DD

==================================Exporting and generating STS certificateStatus : Success

Using config file : /tmp/vmware-fixsts/certool.cfg

Status : SuccessEnter password for [email protected]:

Amount of tenant credentials: 1

Exporting tenant and trustedcertchain 1 to /tmp/vmware-fixstsDeleting tenant and trustedcertchain 1Applying newly generated STS certificate to SSO domain

adding new entry "cn=TenantCredential-1,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local"adding new entry "cn=TrustedCertChain-1,cn=TrustedCertificateChains,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local"Replacement finished - Please restart services on all vCenters and PSCs in your SSO domain

==================================

IMPORTANT: In case you're using HLM (Hybrid Linked Mode) without a gateway, you would need to re-sync the certs from Cloud to On-Prem after following this procedure

==================================

==================================

Note: If receive the following error when trying to run the script:

bash: ./fixsts.sh: /bin/bash^M: bad interpreter: No such file or directory

This error is caused by DOS carriage returns added to the script when copying from a Windows-based text editor. To resolve this problem, run this command and rerun the script:

# sed -i -e 's/\r$//' fixsts.sh

Notes:

If upgraded from vSphere 5.x or 6.0, there may have been multiple STS chains (trustedcertchain) present in TenantCredentials. When the nodes which issued the other chains are no longer accessible, this can cause authorization errors.
Running the script will delete all the old STS chains and replace with a single new chain, resolving the authorization errors caused by the obsolete chains.
If there are multiple certificate chains and no LEAF in the (trustedcertchain) please proceed to regenerate the STS Certificate.

Additional Information

For more information on viewing the STS certificate and determining the expiry date, see Checking Expiration of STS Certificate on vCenter Server.
VMware Skyline Health Diagnostics for vSphere - FAQ

Warning

This script interacts with the VMDIR's database. Take an offline snapshot concurrently for all vCenter Servers and Platform Service Controllers in the SSO domain before running the script. Failing to do so may result in an unrecoverable error and require redeploying vCenter Server.

Notes:

This script should only be run once per SSO domain.
In environments containing Horizon View, see Connection Server unable to accept vCenter thumbprint with an error "There was an error identifying the validity of the server" (67701).

Attachments

fixsts get_app