vCenter Server 7.0.x update fails due to invalid credentials

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• VAMI UI will report the failure as "Exception occured in postInstallHook"

• Depending on the situation, you may receive one of two errors in /var/log/vmware/applmgmt/PatchRunner.log:

• Error 1 ("Failed to register VMware Token Service with Lookup Service"):

2022-06-09 18:55:36,123.123Z vmidentity:Contract INFO vmidentity Failed to register VMware Token Service with Lookup Service.
2022-06-09 18:55:36,124.124Z vmidentity:Contract ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vmidentity:Contract' failed.
...
faultcode: ns0:FailedAuthentication
faultstring: Invalid credentials
faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>Invalid credentials</faultstring></S:Fault></S:Body></S:Envelope>
2022-06-09T16:55:36.129Z ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.

• Error 2 ("An error occurred while starting service 'vsphere-ui'"):

2022-06-09T04:50:45.362Z ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got unhandled exception.
Traceback (most recent call last):
  File "/storage/updatemgr/software-updatexhvajaim/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, ...
service_manager.IllegalServiceOperation: Service cannot be started. Error: Error executing start on service vsphere-ui. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vsphere-ui"
            ],
            "localized": "An error occurred while starting service 'vsphere-ui'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}

• If you encounter error 2, look for another corresponding error in /var/log/vmware/vmon/vmon.log:

2022-06-09T04:50:44.004Z Wa(03) host-7832 <vsphere-ui> Service pre-start command's stderr: Traceback (most recent call last):
2022-06-09T04:50:44.004Z Wa(03)+ host-7832   File "/usr/lib/vmware-vsphere-ui/firstboot/vsphere_ui_prestart.py", line 111, in <module>              
2022-06-09T04:50:44.004Z Wa(03)+ host-7832     raise ex    
...                                                    
2022-06-09T04:50:44.004Z Wa(03)+ host-7832 pyVim.sso.SoapException: SoapException:       
2022-06-09T04:50:44.004Z Wa(03)+ host-7832 faultcode: ns0:FailedAuthentication           
2022-06-09T04:50:44.004Z Wa(03)+ host-7832 faultstring: Invalid credentials    

Environment

VMware vCenter Server 7.0.x

Cause

This failure is due to stale administrator credentials that are left over from the initial installation or a previous major upgrade.

Resolution

This is a known issue and is fixed in VC 7.0U3g

Workaround:
To workaround the issue follow the below steps

1.  Take an offline snapshot (vCenter VM powered off) of vCenter Server ( All nodes in case of Enhanced Linked Mode).
2.  Power on vCenter VM and make sure vCenter UI is accessible and inventory is loaded successfully.
3.  Run commands to clear the SSO credentials

/bin/install-parameter -s vmdir.password
/bin/install-parameter -s vmdir.username

4. Try the vCenter upgrade again.

5. After successful patching, run the following to reset the vmdir.username install parameter

NOTE:  The aforementioned workaround will cause future PNID change operations to fail unless you set the install-parameters for vmdir.username manually (it is not necessary to set vmdir.password):

/bin/install-parameter -s vmdir.username administrator@<your_SSO_domain_here>

Example:

/bin/install-parameter -s vmdir.username [email protected]