AVI upgrade to 31.1.2 pre-check failed with error "Database Error - Invalid Reference"

Products

• VMware Avi Load Balancer

Issue/Introduction

While performing upgrade from AVI Load balancer 30.2.3 to 31.1.2 upgrade pre_check is failing with below error.

This error can be verified from the controller shell.

Step-1: Login to controller Shell

Step-2: Run Command "show upgrade status detail" and check for the below error

{
                    "check_code": "SYSERR_MC_CONFIG_CHK_ERR",
                    "description": "'Export configuration is failed.'",
                    "details": [
                        "Error while checking exported config, output: Traceback (most recent call last):\n  File \"/opt/avi/scripts/config_checker.py\", line 191, in <module>\n    raise Exception('Database Error - Invalid Reference : %s' %all_missing_refs)\nException: Database Error - Invalid Reference : {'FileObject': ['/api/fileobject/fileobject-#####-#####-#####-#####']}\n, error: exit status 1"
                    ],
                    "duration": 4,
                    "end_time": "##-##-#### #######",
                    "start_time": "##-##-#### #######",
                    "state": "UPGRADE_PRE_CHECK_ERROR"
                }

Environment

AVI Load Balancer Version: 30.x & 22.1.x

Cause

• As part of the Upgrade Pre-Checks, AVI Load Balancer perform Object reference Checks on the Avi-Config.
• In this case, the File Object:  System-LocationDB-File is missing from the Database but referred in the GeoDB.

 

• Open avi backup configuration file (.json) with text editior and verify where the fileobject-xxxxx-xxxxx-xxxx UUID is mapped in the configuration.
• In this case File-object was referred under "GeoDB" configuration under "System-LocationDB"

"GeoDB": [
                {
                        "files": [
                                {
                                        "enabled": true,
                                        "file_ref": "/api/fileobject/fileobject-####-####-####-####-####,
                                        "index": 0,
                                        "name": "System-LocationDB",
                                        "vendor": "VENDOR_AVI_DEFINED"
                                },

• Same file-object reference is not present under "FileObject" section. Instead the fileobject reference is difference at "System-LocationDB"

"FileObject" : [

{
                        "name": "System-LocationDB-File",
                        "path": "/var/lib/avi/geo_db/admin/System-LocationDB-File",
                        "tenant_ref": "/api/tenant/?name=admin",
                        "type": "GEO_DB",
                        "url": "/api/fileobject/fileobject-####-####-####-####-####",
                        "uuid": "####-####-####-####-####"
                }
]

• Check if any data import is failed for the object under Controller Leader node directory "/opt/avi/log/initial_data.log" 
• Here the initial creation of the System-LocationDB Object via initial-data.py the import itself is failed for the GeoDB with below error, even the File Object System-LocationDB creation is success.  
• Creation of the GeoDB failed with Error :  api.models_file_object.DoesNotExist: FileObject matching query does not exist.  - even those all the objects are present on the DB. 

[###############] ERROR [nonportal.management.commands.initial_data_import:130] ^[[31mError when importing: data {'path': '/api/geodb', 'method': 'post', 'data': {'name': 'System-GeoDB', 'files': [{'name': 'System-LocationDB', 'index': 0, 'enabled': True, 'vendor': 'VENDOR_AVI_DEFINED', 'file_ref': '/api/fileobject?name=System-LocationDB-File'}, {'name': 'System-CountryDB', 'index': 1, 'enabled': True, 'vendor': 'VENDOR_AVI_DEFINED', 'file_ref': '/api/fileobject?name=System-CountryDB-File'}, {'name': 'System-ISPDB', 'index': 2, 'enabled': True, 'vendor': 'VENDOR_AVI_DEFINED', 'file_ref': '/api/fileobject?name=System-ISPDB-File'}, {'name': 'System-LocationDB_v6', 'index': 3, 'enabled': True, 'vendor': 'VENDOR_AVI_DEFINED', 'file_ref': '/api/fileobject?name=System-LocationDB_v6-File'}, {'name': 'System-CountryDB_v6', 'index': 4, 'enabled': True, 'vendor': 'VENDOR_AVI_DEFINED', 'file_ref': '/api/fileobject?name=System-CountryDB_v6-File'}, {'name': 'System-ISPDB_v6', 'index': 5, 'enabled': True, 'vendor': 'VENDOR_AVI_DEFINED', 'file_ref': '/api/fileobject?name=System-ISPDB_v6-File'}], 'last_modified': 0}} g <avi.rest.request_generator.RequestGenerator object at 0x7fa2dfe6f580> rt_usr avisystemuser tenant ^[[0m
Traceback (most recent call last):
  File "/opt/avi/python/bin/portal/nonportal/management/commands/initial_data_import.py", line 111, in add_object
    rsp = view_func(req, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 332, in get
    raise self.model.DoesNotExist(
api.models_file_object.DoesNotExist: FileObject matching query does not exist.

Resolution

Follow Below steps to resolve the issue

Please navigate to Avi Load Balancer UI --> Templates --> Security --> Geo DB 

• Edit "System-GeoDB"
• Select the "System-LocationDB"
• Click on "Remove" 
• Save the configuration

>> Once This configuration is done, we can proceed further with upgrade