vCenter Update from 8.0 Update 2c (8.0.2.00300) to 8.0 Update 3e (8.0.3.00500) Failed with VAMI Error: 503 - Service Unavailable

Products

• VMware vCenter Server

Issue/Introduction

• The vCenter patching from version 8.0.2.00300 (Update 2c) to 8.0.3.00500 (Update 3e) failed during the startup of multiple services.
• From the vCenter /var/log/vmware/applmgmt/PatchRunner.log file, the service start failure can be seen: 

service_manager.IllegalServiceOperation: Service cannot be started. Error: Error executing start on service vpxd-svcs. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd-svcs"
            ],
            "localized": "An error occurred while starting service 'vpxd-svcs'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}
Service-control failed. Error: {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd-svcs"
            ],
            "localized": "An error occurred while starting service 'vpxd-svcs'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}

YYYY-MM-DDThh:mm:ssZ WARNING root stopping status aggregation...
YYYY-MM-DDThh:mm:ssZ ERROR __main__ Patch vCSA failed

• Checking the /var/log/vmware/vpxd-svcs/vpxd-svcs.log, LDAP authentication errors before the service start failure can be seen:

YYYY-MM-DDThh:mm:ssZ [pool-5-thread-1 [] ERROR com.vmware.cis.lotus.LdapUtils  opId=] Failed to connect to LDAP; uri: ldap://vCenter-FQDN:389
YYYY-MM-DDThh:mm:ssZ [pool-5-thread-1 [] WARN  com.vmware.cis.lotus.LdapConnectionFactory  opId=] Failed to connect to LDAP server at vCenter-FQDN, will retry; attempt:2 of 15, delay:5 sec

Environment

VMware vCenter Server 8.0.x

Cause

The lookup service registrations may contain an SSL trust value that does not match the MACHINE_SSL_CERT on port 443 of the node. Such SSL trust mismatches can occur if the vCenter's machine SSL certificate was recently reset following its expiration. If the updated certificate trust is not propagated to all relevant vCenter services, it can result in service shutdowns.

Resolution

• Take snapshot of the vCenter VM from the respective ESXi host it is residing on.

Note : VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice.

• Use lsdoctor and run the python lsdoctor -t to perform the SSL trust fix of the services.

Additional Information

We can use VDT Tool also to confirm if there are any SSL trust mismatch issues in the lookup service.