Upgrade failed with error "Unable to complete task install_controller"

Products

• VMware Avi Load Balancer

Issue/Introduction

Upgrade failed with error "Unable to complete task install_controller".

 

Cause

- This is happening because of a crash in the config_checker process.
- This process is run as a part of the Install Controller task. This task is basically responsible for booting up the controller VM in the new version.
- The config_checker process performs a sanity check of the configuration.
- However, when that process crashes, the upgrade aborts and the system is restored to the last version.
- The process crash should be visible by navigating to Administration>Support>Crash Reports.
- The stack trace will look like:

Traceback timestamp: 20240925_212600
Command line: /opt/avi/scripts/config_checker.py -c /var/lib/avi/backups//backup_Default-Scheduler_<>.json
Traceback (most recent call last):
  File "/opt/avi/scripts/config_checker.py", line 125, in <module>
    raise Exception('Database Error - Invalid Reference : %s' %all_missing_refs)
Exception: Database Error - Invalid Reference : {'StringGroup': ['/api/stringgroup/stringgroup-xxxxx', '/api/stringgroup/stringgroup-xxxxx'], 'IPReputationDB': ['/api/ipreputationdb/ipreputationdb-xxxxx'], 'BotIPReputationTypeMapping': ['/api/botipreputationtypemapping/botipreputationtypemapping-xxxxx'], 'BotConfigConsolidator': ['/api/botconfigconsolidator/botconfigconsolidator-xxxxx'], 'BotMapping': ['/api/botmapping/botmapping-xxxxx']} 

Resolution

- The crash is happening because the BotDetectionPolicy is referencing the wrong uuids.
- This is a system default object and cannot be deleted.
- The fix would be to correct those references.
- This can be done from the CLI using the commands below:

> configure botdetectionpolicy System-BotDetectionPolicy
> ip_reputation_detector ip_reputation_db_ref System-IPReputation-Webroot-DB
> save
> ip_reputation_detector system_ip_reputation_mapping_ref System-BotIPReputationTypeMapping
> save
> ip_location_detector system_cloud_providers_ref System-BotCloudProviders
> save
> ip_location_detector system_search_engines_ref System-BotSearchEngines
> save
> system_consolidator_ref System-BotConfigConsolidator
> system_bot_mapping_ref System-BotMapping
> save

- Once the references are corrected, the upgrade can be reattempted and it should succeed.