vCenter upgrade failed with error message "Internal error occurs during Import of VMware vSphere Auto Deploy"

Products

VMware vCenter Server

Issue/Introduction

During the vCenter Server upgrade to 9.0, the process fails during the data import phase for VMware vSphere Auto Deploy (vmware-rbd).
In the vCenter UI, the following error is displayed:

Internal error occurs during Import of VMware vSphere Auto Deploy.

In the /var/log/vmware/upgrade/import-upgrade-runner.log, you will see the following errors:

INFO upgrade.states.component_states rbd:Import: INFO autodeploy.upgrade_util Ran the registration script stdout:b'', stderr:b"error: cannot access local variable 'vcId' where it is not associated with a value\n", returncode:159
INFO upgrade.states.component_states rbd:Import: ERROR autodeploy.upgrade_util Unable to re-register autodeploy service after upgrading
INFO upgrade.states.component_states rbd:Import: ERROR autodeploy.autodeploy_upgrader Error occurred while importing the data for autodeploy
...
INFO upgrade.states.component_states rbd:Import: Exception: Unable to re-register autodeploy service after upgrading
INFO upgrade.states.component_states rbd:Import: ERROR __main__ Upgrade Phase 'autodeploy:Import' failed. Exception: Unable to re-register autodeploy service after upgrading

Additionally, the /var/log/vmware/rbd/autodeploy-register.log reports 'vcId' is not associated with a value:

INFO:vc_extension:updating rule extension info
ERROR:autodeploy-register:error: cannot access local variable 'vcId' where it is not associated with a value
Traceback (most recent call last):
  File "/usr/bin/autodeploy-register", line 562, in <module>
    sys.exit(main(sys.argv))
             ^^^^^^^^^^^^^^
  File "/usr/bin/autodeploy-register", line 490, in main
    configpairs.replace(key="DEFAULT_VC", value=vcId)
                                                ^^^^
UnboundLocalError: cannot access local variable 'vcId' where it is not associated with a value

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 9.0.x

Cause

This issue occurs when the Auto Deploy database or configuration state on the source vCenter contains stale, missing, or corrupted registration data. During the upgrade, the autodeploy-register script fails to retrieve or assign a valid vCenter ID (vcId) from the existing configuration database, causing the script to crash and the upgrade phase to halt.

Resolution

NOTE: The following steps involve purging the Auto Deploy database. Only proceed with these steps if the VMware vSphere Auto Deploy feature is NOT actively used in your environment.

To resolve this issue, reset the Auto Deploy database on the source vCenter Server before retrying the upgrade:

1. Take an offline snapshot of the source vCenter Server VM .

2. SSH into the source vCenter Server appliance using the root credentials.

3. Stop the Auto Deploy watchdog service:

service-control --stop vmware-rbd-watchdog

4. Unregister the Auto Deploy service:

autodeploy-register -U -m -a <vCenterFQDN> -u [email protected] -p 80
(Note: Replace <vCenterFQDN> with the actual FQDN of your vCenter Server)

5. Clear the Auto Deploy database and cache files:

rm -rf /etc/vmware-rbd/db/*
rm -rf /storage/autodeploy/cache/*

6. Re-register the Auto Deploy service to generate a clean configuration:

autodeploy-register -R -m -a <vCenterFQDN> -u [email protected] -p 80

7. Start the Auto Deploy watchdog service:

service-control --start vmware-rbd-watchdog

8. Log in to the vCenter Server Appliance Management Interface (VAMI) at https://<vCenterFQDN>:5480.

9. Navigate to Services, locate the Auto Deploy service, stop it, and ensure its Startup Type is set to Manual.

10. Retry the vCenter upgrade to version 9.0.