vCenter client is not accessible and SPS service fails to start

Products

• VMware vCenter Server

Issue/Introduction

• vCenter is reachable by SSH and VAMI, but accessing vSphere client fails with 404 error.
• Manual restart of vCenter services using command: service-control --start --all shows vCenter service stuck at SPS service startup.
• A watch command: watch service-control --status --all from a new ssh session to vCenter shows the vmware-sps service in pending: 
  
  

  Every 2.0s: service-control -- status -- all

  Running:
lookupsvc lwsmd vc-wsla-broker vmafdd vmcad vmdird vmware-cis-license vmware-eam vmware-envoy vmware-envoy-hgw vmware-envoy-sidecar vmware-infraprofile vmware-postgres-archiver vmware-rhttpproxy vmware-stsd vmware-trustmanagement
are-vapi-endpoint vmware-vmon vmware-vpostgres vsphere-ui vtsdb
StartPending:
vmware-sps
Stopped:
applmgmt observability observability-vapi pschealth vlcm vmcam vmonapi vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-content-library vmware-hvc vmware-imagebuilder vmware-netdumper vmware-perfch
vmware-pod vmware-rbd-watchdog vmware-sca vmware-topologysvc vmware-updatemgr vmware-vcha vmware-vdtc vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vstats wcp

  
  
  
• SPS service goes to a pending state during service startup and multiple vCenter services fail to start.
• On the sps logs: /var/log/vmware/vmware-sps/sps.log,  you see error:

YYYY-MM-DDThh:mm:ss.###Z [main] ERROR opId=sps-Main-52###-3# com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl - Failed to retrieve service content
YYYY-MM-DDThh:mm:ss.###Z [main] ERROR opId=sps-Main-52###-3# com.vmware.vim.storage.common.task.retry.CallableRetryDecorator - Caught exception - com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException: Error occurred while retrieving service content

Environment

VMware vCenter 8.x

Cause

The issue is due to bad values in SPS solution user after the vCenter server is restored from backup or unlinked from ELM(Enhanced Linked Mode).

Resolution

Follow the below steps and recreate the SPS solution user:

• Take an online snapshot of vCenter ( If the vCenters are in ELM take Offline snapshots): VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice)
• Recreate fresh solution users Using the 'lsdoctor' Tool with "-u" option.
• Restart all the services with the below command:
  service-control --stop --all && service-control --start --all