Upgrade pre-check is failing with Error "SYSERR_MC_IMAGE_VALIDATION_ERR" / Invalid state for the image

Products

• VMware Avi Load Balancer

Issue/Introduction

•  AVI Load balancer upgrade was failing with pre-check error "SYSERR_MC_IMAGE_VALIDATION_ERR"

Environment

• AVI Load Balancer Version: All Versions

Cause

• AVI Load balancer upgrade fails with pre-check error "SYSERR_MC_IMAGE_VALIDATION_ERR"

+-------------------+--------------------------------------------------------------------------------------------------------------+
| Field             | Value                                                                                                        |
+-------------------+--------------------------------------------------------------------------------------------------------------+
| uuid              | cluster-xxxxxx-xxxxx-xxxx-xxxxxxxxx                                                                         |
| name              | xxxxxxxxxxxxxxxxxxx                                                                                         |
| node_type         | NODE CONTROLLER_CLUSTER                                                                                     |
| upgrade_ops       | UPGRADE                                                                                                     |
| upgrade_readiness |                                                                                                             |
| checks [1]        |                                                                                                             |
|   check_code      | SYSERR_MC_BACKUP_ERR                                                                                        |
|   description     | Inform User to take configuration backup prior to upgrade operations.                                       |
|   details         | Please take the backup before starting the upgrade operations.                                              |
|   state           | UPGRADE_PRE_CHECK_WARNING                                                                                   |
|   start_time      | 2025-07-18 09:23:14                                                                                         |
|   end_time        | 2025-07-18 09:23:14                                                                                         |
|   duration        | 0 sec                                                                                                       |
|                                                                                                                                 |
| checks [2]        |                                                                                                             |
|   check_code      | SYSERR_MC_IMAGE_VALIDATION_ERR                                                                              |
|   description     | Invalid image state across the Cluster members.                                                             |
|   details         | Invalid state for the image 31.1.1-9122-20250127.143602/image-cd13848c-0ae9-489d-87c6-7c361ae4333f.         |
|   state           | UPGRADE_PRE_CHECK_ERROR                                                                                     |
|   start_time      | 2025-07-18 09:23:12                                                                                         |
|   end_time        | 2025-07-18 09:30:57                                                                                         |
|   duration        | 465 sec                                                                                                     |
+-------------------+--------------------------------------------------------------------------------------------------------------+

• On GUI you will notice error in below image 

The following issue came up while preparing to update:

! Please take the backup before starting the upgrade operations.
! Please take the backup before starting the upgrade operations.
! Please take the backup before starting the upgrade operations.
! Please take the backup before starting the upgrade operations.
! Please take the backup before starting the upgrade operations.
✗ Invalid state for the image 31.1.1-9122-20250127.143602/image-cd13848c-0ae9-489d-87c6-7c361ae4333f.

Cause for this error:

• As a process of upgrade we upload the image on Leader node either using UI/CLI.
• Upon successful upload of the image, it will be copied to Follower nodes from leader node with a timeout of 10 mins
• If the image is not getting copied in 10 mins to the followers, then we will hit this issue.

Steps to verify image copy was successful:

Leader Node:

• Run the command to check if there are any errors "show image <image name>

  events[7]                                   |                                                                  |
|   task_name                                 | ImageReplication                                                 |
|   sub_events[1]                             |                                                                  |
|     ip                                      | node2.controller.local                                           |
|     start_time                              | 2025-08-13 13:52:54                                              |
|     end_time                                | 2025-08-13 13:55:41                                              |
|     duration                                | 167 sec                                                          |
|     status                                  | SYSERR_FAILURE
|     message                                 | after maximum number of retries: Failed to copy /host/pkgs/      |
|                                             | <version>/controller.pkg
| tasks_completed                             | 7                                                                |
| total_tasks                                 | 7                                                                |
| progress                                    | 85 percent                                                      |
| start_time                                  | 2025-08-13 13:51:30                                              |
| end_time                                    | 2025-08-13 14:04:41                                              |
| duration                                    | 251 sec                                                          |
| tenant_ref                                  | admin                                                            |
+---------------------------------------------+------------------------------------------------------------------+

• Navigate to path /host/pkgs/<image version>
• Verify the MD5 sum of the controller.pkg and se.pkg

-rw-r--r-- 1 root root  61M Aug 13 13:52 azure.pkg
-rw-r--r-- 1 root root 4.1G Aug 13 13:51 controller.pkg
-rw-r--r-- 1 root root 1.6G Aug 13 13:52 se.pkg
-rw-r--r-- 1 root root  165 Aug 13 13:52 VERSION

• check md5sum using the below command

admin@xx-xx-xx-xx:/host/pkgs/30.2.3-9116-20250330.005843$ md5sum controller.pkg 
87b2e3f514787ca023780dee0e6cca62  controller.pkg

• Compare the md5sum with the md5sum value available in broadcom support portal for this image.

Follower Node:

• Navigate to path /host/pkgs/<image version>
• Verify the size of packages and see if the size and md5sum of the package is matching.

 

Resolution

• Re-upload the image using Leader Controller Node
• Make sure we are having enough network speed to replicate the images to the follower nodes within 10 mins.