Upgrade of vCenter Server Appliance from version 8.0.3.00600 to 8.0.3.00800 fails.
After rolling back to a snapshot taken before the upgrade, vCenter services fail to start.
The vSphere Client (GUI) is inaccessible, showing a 503 Service Unavailable error.
Running service-control --start --all fails with the following error: Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start sps, vpxd-svcs services. Error: Operation timed out
In /var/log/vmware/vpxd/vpxd.log, errors similar to the following are observed: N3Vim5Fault11InvalidName9ExceptionE (Fault cause: vim.fault.InvalidName) Failed to set VAPI security context; N3Vim5Fault12InvalidLogin9ExceptionE (Fault cause: vim.fault.InvalidLogin)
The vmware-sps (Profile-Driven Storage) service logs (/var/log/vmware/vmware-sps/sps.log) show login failures to the vpxd service: VPXD client login failed. Caught exception - com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException: Error while doing login to VPXD service
vCenter 8.x
This issue is caused by a corruption or mismatch in the Secure Token Service (STS) or Machine SSL certificates that occurs during the failed upgrade process. Even after a snapshot rollback, the internal service registrations in the Lookup Service may retain incorrect or mismatched SSL trust information, preventing services like sps and vpxd-svcs from authenticating and starting correctly.
To resolve this issue, the internal certificates must be refreshed and the service trust must be synchronized.
Prerequisites :
Perform an offline snapshot of the vCenter Server Appliance (and any other vCenters in the same SSO domain/Enhanced Linked Mode) before proceeding.
Ensure you have root access to the vCenter Server Appliance via SSH.
Procedure
Reset Certificates: Use the vCert utility or the Certificate Manager to reset the vCenter certificates, ensuring that the STS certificates are refreshed.
Reference: vCert scripted vCenter expired certificate replacement
Fix Service Trust: Download and run the lsdoctor tool to repair the SSL trust mismatch in the lookup service.
Reference: Using the 'lsdoctor' Tool
Review this command before running it.
python lsdoctor.py -t
Restart vCenter Services: Restart all services to apply the changes.
This command will make changes to your system. Review it carefully before running.
service-control --stop --all
service-control --start --all