vCenter Server 8.0 Upgrade Fails with 503 Service Unavailable and Login Timeouts after Snapshot Rollback
search cancel

vCenter Server 8.0 Upgrade Fails with 503 Service Unavailable and Login Timeouts after Snapshot Rollback

book

Article ID: 439200

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Upgrade of vCenter Server Appliance from version 8.0.3.00600 to 8.0.3.00800 fails.

After rolling back to a snapshot taken before the upgrade, vCenter services fail to start.

The vSphere Client (GUI) is inaccessible, showing a 503 Service Unavailable error.

Running service-control --start --all fails with the following error: Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start sps, vpxd-svcs services. Error: Operation timed out

In /var/log/vmware/vpxd/vpxd.log, errors similar to the following are observed: N3Vim5Fault11InvalidName9ExceptionE (Fault cause: vim.fault.InvalidName) Failed to set VAPI security context; N3Vim5Fault12InvalidLogin9ExceptionE (Fault cause: vim.fault.InvalidLogin)

The vmware-sps (Profile-Driven Storage) service logs (/var/log/vmware/vmware-sps/sps.log) show login failures to the vpxd service: VPXD client login failed. Caught exception - com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException: Error while doing login to VPXD service

Environment

vCenter 8.x

 

Cause

This issue is caused by a corruption or mismatch in the Secure Token Service (STS) or Machine SSL certificates that occurs during the failed upgrade process. Even after a snapshot rollback, the internal service registrations in the Lookup Service may retain incorrect or mismatched SSL trust information, preventing services like sps and vpxd-svcs from authenticating and starting correctly.

Resolution

To resolve this issue, the internal certificates must be refreshed and the service trust must be synchronized.

Prerequisites : 
Perform an offline snapshot of the vCenter Server Appliance (and any other vCenters in the same SSO domain/Enhanced Linked Mode) before proceeding.
Ensure you have root access to the vCenter Server Appliance via SSH.
Procedure
Reset Certificates: Use the vCert utility or the Certificate Manager to reset the vCenter certificates, ensuring that the STS certificates are refreshed.

Reference: vCert scripted vCenter expired certificate replacement 

Fix Service Trust: Download and run the lsdoctor tool to repair the SSL trust mismatch in the lookup service.

Reference: Using the 'lsdoctor' Tool

Review this command before running it.

python lsdoctor.py -t

Restart vCenter Services: Restart all services to apply the changes.

This command will make changes to your system. Review it carefully before running.

service-control --stop --all
service-control --start --all