vCenter upgrade failed with error message "Internal error occurs during Import of VMware vSphere Auto Deploy"

Products

• VMware vCenter Server

Issue/Introduction

• During the Stage 2 of the VCSA upgrade, we may see the following error:

• The target vCenter appliance contain the error message "error: local variable 'vcId' referenced before assignment"

 

/var/log/vmware/upgrade/import-upgrade-runner.log

[YYYY-MM-DDTHH:MM:SS]  INFO upgrade.states.component_states rbd:Import: reported message 'Importing VMware vSphere Auto Deploy data...'

[YYYY-MM-DDTHH:MM:SS]  INFO upgrade.states.component_states rbd:Import:

[YYYY-MM-DDTHH:MM:SS]  INFO upgrade.states.component_states rbd:Import: [YYYY-MM-DDTHH:MM:SS]  INFO service_manager Command '[['/bin/service-control', '--status', 'vmware-rbd-watchdog']]' has exit-code='0' and stdout: Stopped:

[YYYY-MM-DDTHH:MM:SS]  INFO upgrade.states.component_states rbd:Import:  vmware-rbd-watchdog

[YYYY-MM-DDTHH:MM:SS]  INFO upgrade.states.component_states rbd:Import:

[YYYY-MM-DDTHH:MM:SS]  INFO upgrade.states.component_states rbd:Import:

[YYYY-MM-DDTHH:MM:SS]  INFO upgrade.states.component_states rbd:Import: stderr:

[YYYY-MM-DDTHH:MM:SS]  INFO upgrade.states.component_states rbd:Import:

[YYYY-MM-DDTHH:MM:SS]  INFO upgrade.states.component_states rbd:Import: [YYYY-MM-DDTHH:MM:SS] INFO service_manager Service vmware-rbd-watchdog: STOPPED

[YYYY-MM-DDTHH:MM:SS]  INFO upgrade.states.component_states rbd:Import: [YYYY-MM-DDTHH:MM:SS] INFO autodeploy.upgrade_util Successfully wrote the XML changes

[YYYY-MM-DDTHH:MM:SS]  INFO upgrade.states.component_states rbd:Import: [YYYY-MM-DDTHH:MM:SS] INFO autodeploy.upgrade_util dbsetup: dbapi_version=2.6.0; sqlite_version=3.38.5

[YYYY-MM-DDTHH:MM:SS]  INFO upgrade.states.component_states rbd:Import: [YYYY-MM-DDTHH:MM:SS] INFO autodeploy.upgrade_util Successfully performed the db changes

[YYYY-MM-DDTHH:MM:SS]  INFO upgrade.states.component_states rbd:Import: [YYYY-MM-DDTHH:MM:SS]  INFO autodeploy.upgrade_util The autodeploy register arguments ['/usr/bin/autodeploy-register', '-R', '-a', 'localhost', '-s', '/etc/vmware-rbd/autodeploy-setup.xml', '-l', '-f', '-t', '300']

[YYYY-MM-DDTHH:MM:SS]  INFO upgrade.states.component_states rbd:Import: didn't have progress. Next report will come after 60 seconds

[YYYY-MM-DDTHH:MM:SS]  INFO upgrade.states.component_states rbd:Import:

[YYYY-MM-DDTHH:MM:SS]  INFO upgrade.states.component_states rbd:Import: [YYYY-MM-DDTHH:MM:SS]  INFO autodeploy.upgrade_util Ran the registration script stdout:b'', stderr:b"error: local variable 'vcId' referenced before assignment\n", returncode:159

 

/var/log/vmware/upgrade/Import_com.vmware.rbd_YYYY_MM_DD_HH_MM.log 

[YYYY-MM-DDTHH:MM:SS]  INFO service_manager Service vmware-rbd-watchdog: STOPPED

[YYYY-MM-DDTHH:MM:SS]  INFO autodeploy.upgrade_util Successfully wrote the XML changes

[YYYY-MM-DDTHH:MM:SS]  INFO autodeploy.upgrade_util dbsetup: dbapi_version=2.6.0; sqlite_version=3.38.5

[YYYY-MM-DDTHH:MM:SS]  NFO autodeploy.upgrade_util Successfully performed the db changes

[YYYY-MM-DDTHH:MM:SS]  INFO autodeploy.upgrade_util The autodeploy register arguments ['/usr/bin/autodeploy-register', '-R', '-a', 'localhost', '-s', '/etc/vmware-rbd/autodeploy-setup.xml', '-l', '-f', '-t', '300']

[YYYY-MM-DDTHH:MM:SS]  INFO autodeploy.upgrade_util Ran the registration script stdout:b'', stderr:b"error: local variable 'vcId' referenced before assignment\n", returncode:159

 

 

var/log/vmware/rbd/autodeploy-register.log

[YYYY-MM-DDTHH:MM:SS] [73629:MainThread]INFO:vc_extension:Using https://vCenter_FQDN:6502 as the managementUrl

[YYYY-MM-DDTHH:MM:SS] [73629:MainThread]INFO:vc_extension:registering servers with full cert and thumbprint -- XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX

[YYYY-MM-DDTHH:MM:SS] [73629:MainThread]INFO:vc_extension:updating rule extension info

[YYYY-MM-DDTHH:MM:SS] [73629:MainThread]ERROR:autodeploy-register:error: local variable 'vcId' referenced before assignment

 

Journalctl

[YYYY-MM-DDTHH:MM:SS]  vCenter_FQDN python[73629]: rbd [73629]autodeploy-register:ERROR:error: local variable 'vcId' referenced before assignment

                                                             Traceback (most recent call last):

                                                               File "/usr/bin/autodeploy-register", line 562, in <module>

                                                                 sys.exit(main(sys.argv))

                                                               File "/usr/bin/autodeploy-register", line 490, in main

                                                                 configpairs.replace(key="DEFAULT_VC", value=vcId)

                                                             UnboundLocalError: local variable 'vcId' referenced before assignment

 

 

• Additionally, in var/log/vmware/rbd/rbd-syslog.log , you will see error message "no such table: vc_servers"

 

[YYYY-MM-DDTHH:MM:SS] err rbd  [22528]rbd-feedback:ERROR:Program exiting due to unknown exception

Traceback (most recent call last):

  File "/usr/bin/rbd-feedback", line 160, in <module>

    sys.exit(main(sys.argv))

  File "/usr/bin/rbd-feedback", line 52, in main

    return rbd_feedback.main(args)

  File "/var/lib/rbd/bin/rbd_feedback.py", line 40, in main

    feedbackServer = feedback.FeedbackServer(extraArgs[0])

  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/waiter/feedback.py", line 48, in __init__

  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/waiter/vc_servers.py", line 28, in findByAddr

  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/dbsetup.py", line 712, in gather

sqlite3.OperationalError: no such table: vc_servers

 

[YYYY-MM-DDTHH:MM:SS] err rbd  [63049]networkutil:ERROR:127.0.0.1 : hostAddr is either loopback or link-local

Environment

vCenter 8.x

Cause

The vc_servers table is missing from the database

Resolution

• To resolve this issue, reset Auto Deploy database by following the steps outlined in KB 374630
• Once completed, restart the vCenter upgrade from Stage 1.