vCenter Upgrade fails with error "Failed to register Auto Deploy"

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

vCenter upgrade fails with error "Failed to register Auto Deploy"

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

• Auto-deploy service fails to register on the newly deployed vCenter and eventually gets timed out upon retries.
• Auto-deploy depends on VPXD to register.
• The VPXD service is busy on the source vCenter and therefore the service registration on the new deployed vCenter is failing. 

VPXD service can be busy due to:

1. Repeated login failures 
2. Repeated querying of hosts if there is a connectivity issue between vCenter and host(s)

In /var/log/firstboot/autodeploy-firstboot.py_******_stderr.log

YYYY-MM-DDTHH:MM:SS,508  Service operation failed: {{
        {
            "id": "install.autodeploy.register.failed",
            "translatable": "Failed to register Auto Deploy",
            "localized": "Failed to register Auto Deploy"
        }
    ],
    "componentKey": null,
 "problemId": null, "resolution": null }} Traceback (most recent call last): File "/usr/lib/vmware-rbd/firstboot/autodeploy-firstboot.py", line 576, in _registerRBD invoke_command(command, quiet=True) File "/usr/lib/vmware/site-packages/cis/utils.py", line 371, in invoke_command raise InvokeCommandException(errStr='Stderr: %s' % stderr) cis.exceptions.InvokeCommandException: { "detail": [ { "id": "install.ciscommon.command.errinvoke", "translatable": "An error occurred while invoking external command : '%(0)s'", "args": [ "Stderr: /usr/bin/autodeploy-register:249: DeprecationWarning: This method will be removed in future versions. Use 'parser.read_file()' instead.\n parser.readfp(f)\nerror: cannot connect to vCenter -- The read operation timed out\n" ], "localized": "An error occurred while invoking external command : 'Stderr: /usr/bin/autodeploy-register:249: DeprecationWarning: This method will be removed in future versions. Use 'parser.read_file()' instead.\n parser.readfp(f)\nerror: cannot connect to vCenter -- The read operation timed out\n'"}

vpxd logs on the source vCenter 

YYYY-MM-DDTHH:MM:SSZ error vpxd[06222] [Originator@6876 sub=User opID=7a69eae5] Failed to authenticate user <******>
YYYY-MM-DDTHH:MM:SS.Z info vpxd[06190] [Originator@6876 sub=vpxLro opID=sps-Main-549620-716-de16-67] [VpxLRO] -- BEGIN lro-6345 -- AuthorizationManager -- vim.AuthorizationManager.hasPrivilegeOnEntities -- 52137186-63f0-44ec-4230-f714fbed1ef7(52425783-3515-4b56-d0d1-fff1bef7b9b8)
YYYY-MM-DDTHH:MM:SS info vpxd[06190] [Originator@6876 sub=vpxLro opID=sps-Main-549620-716-de16-67] [VpxLRO] -- FINISH lro-6345
YYYY-MM-DDTHH:MM:SS info vpxd[05266] [Originator@6876 sub=SsoClient opID=39347598] Successfully acquired token: SamlToken [subject={Name: ****; Domain:*****}, groups=[{Name: Users; Domain:********}, {Name: Domain Users; Domain:*******}, {Name: ibm-relay; Domain:********}, {Name: Everyone; Domain:vsphere.local}], delegationChain=[], startTime=YYYY-MM-DD HH:MM:SS., expirationTime=YYYY-MM-DD HH:MM:SS, renewable=false, delegable=false, isSolution=false,confirmationType=0]
YYYY-MM-DDTHH:MM:SS info vpxd[05266] [Originator@6876 sub=vpxLro opID=39347598] [VpxLRO] -- FINISH lro-5802
YYYY-MM-DDTHH:MM:SS error vpxd[10584] [Originator@6876 sub=[SSO] opID=3e47efa] [UserDirectorySso] AcquireToken exception: N9SsoClient22CommunicationExceptionE(Operation timed out)
--> [context]zKq7AVECAAAAACrfOAETdnB4ZAAAEFcrbGlidm1hY29yZS5zbwAAJEUbAOCwGAFZPgJsaWJzc29jbGllbnQuc28AAZNIAgHwEQICpjb2dnB4ZAAC6gyfAkkqoIMkrQEBbGlidmltLXR5cGVzLnNvAAJpHHMC0iByAoMucgIKiXIAlb0jAF7AIwDNrCsE1HMAbGlicHRocmVhZC5zby4wAAXtjw5saWJjLnNvLjYA[/context]

Number of login failures reported

grep "Failed to authenticate user" vpxd-????.log | wc -l 
309775

Resolution

1. Disable the user reported in the logs from the vCenter.

2. Increase the registration timeout value for the auto-deploy service on the new deployed vCenter. 

• Take a SSH session to the new deployed vCenter 
• Login with root credentials 
• Enable shell - shell.set --enable true 
• Navigate to location -  cd /var/lib/rbd/bin
• Edit the file - vi autodeploy-register
• Find the keyword default under block "Maximum number of seconds spent trying to register with vCenter"

 

• Hit i to edit
• Change the default value to 900
• Let the upgrade continue