vCenter Server UI showing "No Healthy Upstream" error and vpxd-svcs and vpxd services are failing to start

Products

• VMware vCenter Server

Issue/Introduction

• When checking vCenter Server services, we can see all but vpxd, vpxd-svcs and sps are up and running 
• When we check STS certificate, it is showing as valid and in date through the command line, this can be done using the vCert script: vCert - expired certificate replacement script
• When checking other certs such as Machine SSL and Solution Users, all are in date, vCert can be used to check these, or by running below command in an SSH session to vCenter: 

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | grep -i "not after"; done;  

• In /var/log/vmware/vpxd-svcs.log we see errors similar to below: 

"Error communicating to the remote server http://localhost:####/sts/system-STSService
javax.xml.ws.WebServiceException: java.io.IOException: Error writing to server
        at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage"

• In /var/log/vmware/sso/vmware-identity-sts-default.log   may see errors similar to below: 

"ERROR sts-default [##:Thread-#] [CorId= OpId=] [com.vmware.identity.util.VcTrustCache] Refresh thread failed to retrieve Vctrusts."

Exampe:

 

 

Environment

vCenter Server 7.0

vCenter Server 8.0 

 

Cause

Issue with STS certificate authentication is causing vpxd and vpxd-svcs to be down and fail to start back up

Resolution

**Ensure to have valid snapshots/backups (offline snapshots of all nodes in Enhanced Linked Mode) are completed prior to making changes

Please follow below KB to use vCert tool to update STS certificate and restart services on all nodes: 

vCert - expired certificate replacement script