VCF Operations Upgrade /Patch task stuck in progress

Products

• VCF Operations

Issue/Introduction

• During a VMware Cloud Foundation (VCF) Operations upgrade/patch, the task hangs and remains stuck in an "In Progress" state within the UI.
  

Environment

• VCF Operations 9.1

Cause

This issue occurs because the underlying package manager (pakManager) fails to properly register the completion of the cleanup phase. Even though the upgrade may have applied successfully, the .results JSON file for the upgrade package retains null or empty values for the cleanup status. Because these values do not show as completed, the VCF Fleet lifecycle management system cannot mark the upgrade task as finished, causing it to hang in the UI.

 

This can be verified by reviewing the casa.log and the corresponding .results file.

# /storage/log/vcops/log/casa/casa.log 

INFO casa 744161 [ops@4413 threadId="30324" threadName="ajp-nio-127.0.0.1-8011-exec-29" requestId="3D0007qE"] [com.vmware.vcops.casa.support.subprocess.GeneralCommand.execute:254] - Command '/usr/bin/sudo -n /usr/lib/vmware-python-3/bin/python /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/vcopsPakManager.py --action query_pak_status --pak vRealizeOperationsManagerEnterprise-910040025541550 --json --force_content_update false' completed successfully with exit code '0'.

Notice that cleanup_exit_code and cleanup_result are empty "no results"

#/storage/db/pakRepoLocal/vRealizeOperationsManagerEnterprise-910040025541550/vRealizeOperationsManagerEnterprise-910040025541550.results

...
    "certs_renew_exit_code": "0",
    "certs_renew_result": "Completed successfully",
    "cleanup_exit_code": "",
    "cleanup_result": "no results",
...

Resolution

Note: Only perform the steps if the upgrade/patch has completed successfully on all analytical nodes and cloud proxies. Validate the same from Admin UI page of VCF Operations.

 

To resolve this issue, you must manually update the results file to reflect a successful cleanup.

1. Take a SNAPSHOT of the VCF Operations.
2. SSH into your primary node as root.
3. Navigate to the local package repository directory:

   cd /storage/db/pakRepoLocal/vRealizeOperationsManagerEnterprise-<respective_version>/

4. Backup the existing .results file:

   cp -p vRealizeOperationsManagerEnterprise-<respective_version>.results vRealizeOperationsManagerEnterprise-<respective_version>.results.bak

5. Edit the file:

   vi vRealizeOperationsManagerEnterprise-<respective_version>.results

6. Locate the cleanup values and update them to indicate a successful completion.

   • Change from:

     "cleanup_exit_code": "",
     "cleanup_result": "no results",

   • Change to:

     "cleanup_exit_code": "0",
     "cleanup_result": "Completed successfully",

7. Save and exit the file (Esc + :wq!).
8. VCF Fleet will poll (~ 2 mins) the correct values and update the task status in the UI to completed.

Additional Information

• Ensure you are modifying the directory and filename corresponding to the exact PAK version used in your environment.
• If the task does not get updated within 10 minutes, delete the Fleet Build /Upgrade and SDDC Build/Upgrade pods from the Control Plane of VCF Services Runtime  Cluster. Refer: Fleet Manager LCM or SDDC Manager LCM Tasks appear to be stuck or hung.