Avi Controller Upgrade Fails with UPGRADE_FSM_ABORTED Due to Patch Version Mismatch on One Controller Node
search cancel

Avi Controller Upgrade Fails with UPGRADE_FSM_ABORTED Due to Patch Version Mismatch on One Controller Node

book

Article ID: 422851

calendar_today

Updated On:

Products

VMware Avi Load Balancer

Issue/Introduction

The Avi Controller upgrade fails after the controller reboot during a cluster upgrade.

The following behavior is observed:

  • Upgrade status shows UPGRADE_FSM_ABORTED

  • Error indicates ["Cluster nodes need to be homogeneous regarding memory, CPU, disk, version, patch version, and FIPS mode."]

  • The patch is applied on the leader and follower nodes, but one follower node reports the patch version as Empty, resulting in a patch version mismatch across the controller cluster.

  • Errors are logged in /var/lib/avi/log/upgrade-coordinator.log on the Controller Leader node (check both current and previous partitions).

  • Due to this mismatch in patch status across controller nodes, the upgrade does not complete.

Environment

  • Avi Load Balancer

Cause

  • Patch image transfer from the leader to one follower controller node fails due to a connectivity interruption.

  • The patch package on the affected follower node remains incomplete or corrupted.

  • The connection lost between controller nodes during patch image transfer is logged in /var/lib/avi/log/remote_task_manager.INFO on the Controller Leader node.

Resolution

  • After the controller upgrade fails, identify the controller node where the patch version shows as empty by reviewing /var/lib/avi/log/upgrade-coordinator.log

  • Log in to the affected controller node and verify the patch status in the old partition; if the patch version shows as null, the patch is not applied.

  • Validate the MD5 checksum of the controller patch package located at /host/pkgs/<patch_directory>/ on the follower controller node.

  • Run the same MD5 command on the leader controller node and compare the values; if the checksums do not match, the patch package on the follower node is corrupted.
  • Remove the existing patch package from the Avi Controller GUI on the leader node, upload a fresh patch package, and restart the controller upgrade process to complete it successfully.
Powered by Wolken Software