Some of the virtual services go down during an upgrade, leading to traffic disruptions.

Products

• VMware Avi Load Balancer

Issue/Introduction

• VS placement may fail for some VSs with the error "Failed to add VS to the interface".
  
• The VSs affected by this will be down and not processing traffic.

Environment

• Avi NSX-T deployments with SEs created on versions lesser than 30.2.1.

Cause

• The customer environments, where the "nsxt_no_hotplug" is set as False in the SE data vNics, might experience issues like this because of a bug in Cloud Connector.
• All SEs created before v30.2.1 will have the "nsxt_no_hotplug" set to False by default. Hence, they are vulnerable to this.
• The cloud connector ends up setting a valid NIC as delete pending due to non-deterministic behavior of the encoding json library and removing it in the subsequent delete vNic request from its runtime, causing traffic disruption after an upgrade.

Resolution

• Bug ID: AV-237208
• Details: Some of the virtual services go down during an upgrade, leading to traffic disruptions.
• Fix Versions: 30.2.4+, 31.1.2, 31.2.1
• Release Notes: 30.2.4 Release Notes
• Workaround: Run the script fix_sevm_runtime.py shipped with the latest version, 31.2.1+, of the controller BEFORE upgrading the controller.
  • To execute the script, SSH into the controller.
  • The script can be run as using the command below:

    $ sudo -i
    # cd /opt/avi/scripts
    # python3 fix_sevm_runtime.py -h
    Usage: fix_sevm_runtime.py [options] arguments
    
    Options:
      -h, --help         show this help message and exit
      --se-name=SE_NAME  Name of the Service Engine you want to fix.
      --fix              Fix the SE VM Runtime configuration.
      --dry-run          Do a dry run and list the fixed SE VM Runtime without
                         saving. (default: True)
    
    # python3 fix_sevm_runtime.py --se-name=<se-name> [--fix]

 

• Note: If you are running into this issue on a version less than 31.2.1, please reach out to support and we can assist you accordingly.