Upgrade hangs at step 8 of 14 "Applying operating system update" in Aria Operations

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Introduction

Upgrades to Aria Operations 8.18.x may stall indefinitely at step 8 of 14. This behavior is typically observed when pre-upgrade validation checks fail due to incorrect node status configuration. Users may see the upgrade progress bar remain stationary for several hours while "Applying operating system update" is displayed.

Symptoms

The new_validate_stderr.log file (located in the PAK repository directory) contains the following error:
INFO - Pre-upgrade checks "['node_online_check']" got failed

Additional failure details can be found in the associated validation JSON:
/storage/db/pakRepoLocal/vRealizeOperationsManagerEnterprise-[ID]/vRealizeOperationsManagerEnterprise-[ID]_validate.json

Environment

Aria Operations 8.18.x

Cause

The upgrade process is unable to verify the node status because the primary node is not identified correctly within the casa.db.script file.

Resolution

1. Take a snapshot of the environment following the best practices in https://knowledge.broadcom.com/external/article/342576/. 
2. From **Admin UI** take the cluster offline and perform an offline snapshot of the cluster nodes.
3. Log into each node via SSH as the "root" user.
4. Navigate to the following directory '/storage/db/casa/webapp/hsqldb/'.

5. Edit the casa.db.script file to ensure the primary and replica nodes are correctly flagged:

   • For the Primary node: "is_admin_node":true,"ip_address":"####"

   • For the Replica node: "is_admin_node":false,"ip_address":"####"

6. Bring cluster back online from the **Admin UI**.
7. Re-initiate the upgrade process.

*Note: If the issue persists in Aria Operations 8.18.7 or higher, ensure you are using the latest upgrade PAK file. See Download Broadcom products and software for the latest releases.