"no healthy upstream" error when attempting to access the vCenter Server with vSphere Client

Products

• VMware vCenter Server

Issue/Introduction

• Following error is observed when trying to access the vCenter Server UI
  "no healthy upstream" 
  
  
• Many services such as vmware-stsd, vsphere-ui, and vmware-vpxd are stopped
  
  
• manually start services using "service-control --start --all" but cannot start stopped services
• error observed in /var/log/vmware/vapi/endpoing.log
  

  YYYY-MM-DDThh:mm:ss.xxx+yyy ERROR | state-manager1            | DefaultStateManager            | Unexpected error while initializing endpoint runtime state.
  com.vmware.provider.VecsException: Native platform error [code: 4312][Native platform error [code: 4312][Opening store 'vsphere-webclient' failed. [Server: __localhost__, User: __localuser__]]]
          at com.vmware.provider.VecsKeyStoreEngine.engineGetKey(VecsKeyStoreEngine.java:215)
  ：
          at java.lang.Thread.run(Thread.java:748)
  Caused by: com.vmware.identity.vecs.VecsGenericException: Native platform error [code: 4312][Opening store 'vsphere-webclient' failed. [Server: __localhost__, User: __localuser__]]
          at com.vmware.identity.vecs.VMwareEndpointCertificateStore.BAIL_ON_ERROR(VMwareEndpointCertificateStore.java:491)
  

  
  
  
• error observed in /var/log/vmware/vmafdd/vmafdd.log
  

  YYYY-MM-DDThh:mm:ss.xxx [vmafdd][ERROR] ERROR! [VecsIpcGetEntryByAlias] is returning  [4312]

  
  
  
• Unable to list solution user certificates in vecs
  

  #/usr/lib/vmware-vmafd/bin/vecs-cli store list
  MACHINE_SSL_CERT
  TRUSTED_ROOTS
  TRUSTED_ROOT_CRLS

Environment

vCenter Server 8.x

Cause

This can occur either VECS (VMware Endpoint Certificate Store) corrupted or deleted.

Resolution

Restore vCenter Server from backup

Additional Information

Using vCert tools vCert - Scripted vCenter Expired Certificate Replacement to check certificates