vCenter Server 7.0.x update fails due to invalid credentials

Products

• VMware vCenter Server

Issue/Introduction

• VAMI UI will report the failure as "Exception occured in postInstallHook"

• Depending on the situation, one of two errors are seen in /var/log/vmware/applmgmt/PatchRunner.log:

  • Error 1 ("Failed to register VMware Token Service with Lookup Service"):
    
    YYYY-MM-DDTHH:MM.SSSZ vmidentity:Contract INFO vmidentity Failed to register VMware Token Service with Lookup Service. YYYY-MM-DDTHH:MM.SSSZ vmidentity:Contract ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vmidentity:Contract' failed.
...
faultcode: ns0:FailedAuthentication
faultstring: Invalid credentials
faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>Invalid credentials</faultstring></S:Fault></S:Body></S:Envelope> YYYY-MM-DDTHH:MM.SSSZ ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.

• • Error 2 ("An error occurred while starting service 'vsphere-ui'"):
    
    YYYY-MM-DDTHH:MM.SSSZ ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got unhandled exception.
Traceback (most recent call last):
  File "/storage/updatemgr/software-updatexhvajaim/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, ...
service_manager.IllegalServiceOperation: Service cannot be started. Error: Error executing start on service vsphere-ui. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vsphere-ui"
            ],
            "localized": "An error occurred while starting service 'vsphere-ui'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}

• • If error 2 is seen, look for another corresponding error in /var/log/vmware/vmon/vmon.log:
    
    YYYY-MM-DDTHH:MM.SSSZ Wa(03) host-#### <vsphere-ui> Service pre-start command's stderr: Traceback (most recent call last): 
YYYY-MM-DDTHH:MM.SSSZ Wa(03)+ host-#### File "/usr/lib/vmware-vsphere-ui/firstboot/vsphere_ui_prestart.py", line 111, in <module>               
YYYY-MM-DDTHH:MM.SSSZ Wa(03)+ host-#### raise ex                                           
YYYY-MM-DDTHH:MM.SSSZ Wa(03)+ host-#### pyVim.sso.SoapException: SoapException:        
YYYY-MM-DDTHH:MM.SSSZ Wa(03)+ host-#### faultcode: ns0:FailedAuthentication            
YYYY-MM-DDTHH:MM.SSSZ Wa(03)+ host-#### faultstring: Invalid credentials  
    

Environment

VMware vCenter Server 7.0.x

Cause

This failure is due to stale administrator credentials that are left over from the initial installation or a previous major upgrade.

Resolution

This is a known issue and is fixed in VC 7.0U3g.

Workaround:

1.  Take an offline snapshot (vCenter VM powered off) of vCenter Server ( All nodes in case of Enhanced Linked Mode).
2.  Power on vCenter VM and make sure vCenter UI is accessible and inventory is loaded successfully.
3.  Run the following commands to clear the SSO credentials:

/bin/install-parameter -s vmdir.password
/bin/install-parameter -s vmdir.username

4. Attempt the vCenter Server upgrade again.
5. After successful patching, run the following to reset the vmdir.username install parameter

NOTE:  The aforementioned workaround will cause future PNID change operations to fail unless you set the install-parameters for vmdir.username manually (it is not necessary to set vmdir.password):

/bin/install-parameter -s vmdir.username administrator@<SSO_domain>

Example:

/bin/install-parameter -s vmdir.username [email protected]