Avi upgrade fails/aborts on migrate config task due to non-existent network reference
Article ID: 432323
Updated On:
Products
Issue/Introduction
In some scenarios the upgrade may fail/abort during the migrate configuration task. When this occurs the upgrade will rollback and you will see the state of the upgrade set to "UPGRADE_FSM_ABORTED"
Ex.
show upgrade status
+----------------------+--------+---------------+-----------------------------+-----------+-----------------------------+-------+
| Name | Tenant | Cloud | State | Operation | Image | Patch |
+----------------------+--------+---------------+-----------------------------+-----------+-----------------------------+-------+
| controller | admin | - | UPGRADE_FSM_ABORTED : (51)% | UPGRADE | 22.1.2-9086-20220928.053702 | - |You can use the following CLI command to get more details.
show upgrade status detail filter controllerThe command output will provide you with the reason of the upgrade failure and timestamp of the error.
| state | UPGRADE_FSM_ABORTED |
| last_changed_time | Thu Mar 5 20:30:40 2026 ms(7948592) UTC |
| reason | UC::[Thu Mar 5 20:25:07 2026]Error while running task:MigrateConfig |
| | Unable to c |
| | omplete task migrate_config on ['node2.controller.local']. |
| | Traceback (most recen |
| | t call last): |
----SNIP---
| | Exception: Unable to complete task mi |
| | grate_config on ['node2.controller.local']. |
| | .::UC |In a controller cluster each node is mapped as node1, node2, and node3 internally. You can determine the controller mapping with the CLI command below:
show cluster | grep 'name'Typically the upgrade status command will provide you with the configuration object that is causing the failure, however, in this case we need to look at the upgrade logs to identify it.
Once you have determined controller node in which the error occurred you will need to navigate to the previous file system partition and look in the upgrade-coordinator.log file.
Steps:
- ssh to the controller node with the admin user and enter "sudo -i"
- Identify the current file system partition the command below. In the output you will see either "subroot=/root1" or "subroot=/root2."
cat /proc/cmdline - Change directorie to the previous partition.
Example:cd /host/<previous_partition>/var/lib/avi/log/
cd /host/root2/var/lib/avi/log/You can also use the symbolic link for the previous partition.
cd /host/prev/var/lib/avi/log - View the upgrade-coordinator file with vi or vim editor. In the file search for key words "Migrate configuration from previous to current version"
vi upgrade-coordinator.logIn the exception you will find the cause of the failure.
UC::Migration failed for object of type Cloud, with uuid cloud-UUID at function upgrade_Cloud for version 30.2.1 with exception: ("'NoneType' object has no attribute 'get'",) ::UC
Other Symptoms:
If there are overrides to the SE management network or if you scoped out the ESX cluster, ESX hosts, or datastore on SE Groups and these objects were removed from the vCenter DC the migration of the vCenter cloud will also fail and the upgrade will fail/abort with the same error.
When the object no longer exists the SE Group will populate/present the object as a UUID instead of the name of the object in the GUI.
Example:
Environment
Affects upgrades to Version(s):
30.2.x
31.1.x
31.2.x
Cause
The upgrade failure was due to a non-existent network reference in an existing cloud.
The upgrade coordinator log provided the UUID of the offending cloud object. When checking the network object references in the cloud configuration it was determined the management network set in the cloud no longer existed in the network inventory Avi obtains from the DC. By design in a vCenter cloud, Avi will inventory the DC which includes the networks/portgroups.
You can view the configuration of the cloud via the configuration backup with the jq command below or via CLI.
jq '.Cloud[]|select(.uuid == "cloud-UUID")|.vcenter_configuration' <Avi_Backup_Configuration_File>CLI:
show cloud <CLOUD_NAME>Please take note of the management_network field and verify this network exists under Administration > Cloud Resources > Networks > select the cloud in the dropdown.
You can also verify if this network exists via the configuration backup with the jq command below. Use the management_network UUID value from the previous jq command above.
Example of the managment_network value:
"management_network": "/api/vimgrnwruntime/dvportgroup-12340-cloud-UUID"Use the network UUID in the command.
jq '.Network[] | select(.uuid | contains("dvportgroup-12340")) | .name' <Avi_Backup_Configuration_File>If you receive no results then the network does not exist in the Avi controller and vCenter inventory.
Please also check all your SE Groups and ensure there are no nonexistent management networks. ESX clusters, ESX hosts, or data stores configured.
Resolution
Starting 31.1.1 the dry_run option/feature was introduced to perform a system upgrade while the system stays in the current version. This feature will help identify configuration migration issues before performing the upgrade of the system.
Upgrading Avi Load Balancer Controller using UI
Workaround(s):
Update the affected cloud with the correct/existing management network reference.
Remove or correct all affected SE Groups.
Feedback
