Upgrade fail due to configuration migration issues with invalid cloudconnector user ref in VMware AVI Load balancer

Products

• VMware Avi Load Balancer

Issue/Introduction

Upgrade fails during configuration migration due to an invalid cloudconnector user ref in backup configuration.

Go to shell mode on Controller Node

# shell
> show upgrade status details filter controller
Error: Unable to complete task migrate_config on ['node.controller.local']

# Logs to check on Controller Node
/var/lib/avi/log/upgrade-coordinator.log
Unable to complete task migrate_config on ['node.controller.local']

# Example log:
/var/lib/avi/log/task_journal/upgrade_import_config.json
output: Traceback (most recent call last): File "/opt/avi/scripts/config_checker.py", line 125, in <module> raise Exception('Database Error - Invalid Reference : %s' %all_missing_refs) Exception: Database Error - Invalid Reference : {'CloudConnectorUser': ['/api/cloudconnectoruser/cloudconnectoruser-#######-#######-#######-#######']} , error: exit status 1

Environment

VMware Avi Load Balancer

Version: 22.1.x

Cause

This error is caused by a broken reference in Backup Configuration. The cloudconnector user referred by the backup configuration was deleted.

Example:

Under /var/log/nginx/portal.access.log logs, you can see the user was deleted.

[pid: #####|app: 0|req: 59/654590] ##.##.##.## () {66 vars in 1400 bytes} [Wed MM DD 08:23:46 YYYY] DELETE /api/cloudconnectoruser/cloudconnectoruser-######-######-######-######-###### => generated 0 bytes in 82 msecs (####.0 204) 4 headers in 147 bytes (1 switches on core 1)

Resolution

1. Get the cloud connector username:
2. Log in to the controller Shell and identify the username against the cloudconnectoruser-#########-####-####-####-########### UUID.

3. Login: admin
   Password:
   
   [admin: ##-##-##-##]: > show cloudconnectoruser
   +------+---------------------------------------------------------+
   | Name | UUID                                                    |
   +------+---------------------------------------------------------+
   | test | cloudconnectoruser-#########-####-####-####-########### |
   | NSX  | cloudconnectoruser-#########-####-####-####-########### |
   | vc   | cloudconnectoruser-#########-####-####-####-########### |
   +------+---------------------------------------------------------+

4. On the controller GUI.
5. Go to Administration ==> User Credentials ==> User Credentials ==> Search the user.
6. Select the user and delete.
7. 
8. Retry the upgrade.