Avi Load Balancer Upgrade Failure from 22.1.6 to 30.2.3 Due to Invalid Service File Format

Products

• VMware Avi Load Balancer

Issue/Introduction

• An upgrade of the Avi Controller system from version 22.1.6 to 30.2.3 failed, resulting in the Controller remaining at version 22.1.6 while one of the Service Engines (SEs) successfully upgraded to 30.2.3. This version mismatch prevented the SE from connecting to the Controller, blocking further operations and impacting the customer.

Environment

• Avi Controller Version: 22.1.6 (attempting upgrade to 30.2.x version)
• Deployment Type: Potentially bare-metal (LSC - Linux Server Cloud) environments

Cause

• The upgrade failure occurs because the Avi Controller on the bare-metal host uses an old or invalid service file format. Specifically, the avicontroller service file includes the parameter controller_podman_run_params instead of the expected docker_run_params= or the standard podman run/docker run keywords.

   

• During the upgrade, the Avi upgrade script checks the service file parameters to identify and update the Controller image. Due to the unexpected format (controller_podman_run_params), the script fails to correctly detect and update the Controller image to version 30.2.3, causing the Controller to remain on 22.1.6. This mismatch—where an SE runs a higher version (30.2.3) than the Controller (22.1.6)—results in the SE being unable to connect.

   

• It because of old Ansible Jinja template format suspected to generate this outdated/invalid service file structure.

• Example of problematic parameter:
  

  controller_podman_run_params="--name=avicontroller -p 5098:5098 -p 80:80 -p 443:443 -p 8443:8443 -p 5054:5054 -p 161:161/udp -d --privileged -e CONTAINER_NAME=avicontroller -e MANAGEMENT_IP=

Resolution

Recommended Solution for New Deployments (and to prevent future issues):

• Use avi_baremetal_setup.py for Controller Deployment:

  • It is highly recommended to use the avi_baremetal_setup.py script for new Controller deployments. This script ensures the correct, on-the-fly generation of the Controller service file, adhering to the expected format.

  • For Avi LSC (Large Scale Controller) deployments, avi_baremetal_setup.py has been migrated to Python 3 in version 30.2.3. Therefore, for deployments with 30.2.3 and later, ensure you use python3 (Python version >= 3.7) to run avi_baremetal_setup.py.

  • Refer to the official Avi Load Balancer System Requirements for the specific Python version required for your target Controller version: https://techdocs.broadcom.com/us/en/vmware-security-load-balancing/avi-load-balancer/avi-load-balancer/30-2/vmware-avi-load-balancer-installation-guide/preparing-for-installation/system-requirements.html

• Modify avicontroller Service File (if necessary for existing deployments):

  • To prevent this issue in future upgrades on existing deployments that might have this old format, it is advised to update the /hostroot/usr/sbin/avicontroller file.

  • Replace instances of controller_podman_run_params with docker_run_params=.

  • CAUTION: Do not make any changes to the avicontroller service file to replace docker.sock with podman.sock without consulting the Avi Support Team. Incorrect modifications can severely impact Controller functionality.

Workaround:

• Rollback to Consistent State: Due to the SE being on a higher version than the Controller, a direct rollback was complicated. The following steps were used to bring the system to a consistent state (22.1.6) for a fresh upgrade attempt:
  
  1. Unblock Controller Upgrade State: Execute the update_upgrade_usi_info.py script to update the status information for the Service Engine Group (SEG) to unblock the current state and allow for a rollback.
  
  

  python3 /opt/avi/scripts/upgrade/update_upgrade_usi_info.py --update --upgrade-usi-uuid <SE-Group-UUID>

  2. Resolve connect_se.py Issues: If manually connecting SEs using connect_se.py script fails with "Fatal error: Needed to prompt for a connection or sudo password...", ensure that SSH connectivity without interactive password prompts (e.g., using SSH keys) is configured between the Controller and SEs.
  
  3. Downgrade Service Engines (SEs): Once the Controller is unblocked and connectivity is established, manually execute the upgrade_se.sh script on the affected SEs to downgrade them to the Controller's version (22.1.6).
  
  

  /opt/avi/scripts/upgrade_se.sh -t 22.1.6-9191-20240227.161336 -c node1.controller.local -i /vol/pkgs/22.1.6-9191-20240227.161336/se_docker.tgz -f False -b -pv 2p8 -n /vol/pkgs/22.1.6-9004-2p8-20250321.092409/se_patch.pkg

  
  4. Complete System Rollback: Ensure the entire system is consistently rolled back to the desired 22.1.6-2p8 version.

Additional Information

• Do Not Use Internal Scripts Without Guidance: The scripts /opt/avi/scripts/upgrade_se.sh and /opt/avi/scripts/update_upgrade_usi_info.py are internal tools. Do not use them unless explicitly instructed by Avi Support and after a clear understanding of the issue and the current state of the system. Misuse of these scripts can lead to further complications.
• The issue highlights the importance of using the correct and updated deployment tools/templates provided by Avi Networks to ensure smooth upgrades and operational stability.