VCF Operations upgrade stuck at "Run Component Upgrade Prechecks"

Products

• VCF Operations

Issue/Introduction

During a VCF Operations component upgrade (e.g. 9.1.0.0100 → 9.1.0.0400), the upgrade hangs indefinitely at Run Component Upgrade Prechecks, reported in the UI as waiting on LCM legacy node verification.

Environment

VCF Operations 9.1.x

Cause

The CaSA service caches Service Registry entries in memory. The CaSA service populates the registry entries only once during construction. The upgrade issue occurs due to the following sequence:

• A registry entry is deleted or updated on the admin node. For example, the ops-lcm entry represents the legacy OPS LCM 9.0 component retiring during migration.

• The admin node syncs the change to peer nodes using the on-disk services.json file.

• The in-memory cache on non-admin nodes fails to invalidate the old data.

• The non-admin nodes continue serving the stale, pre-delete state until an administrator restarts the CaSA service.

• The precheck workflow queries the stale state and never observes the expected post-delete condition.

• Because the precheck workflow cannot observe the post-delete condition, the upgrade process hangs.

Resolution

1. Take a snapshot of VCF Operation cluster
2. Take SSH session to all cluster nodes using root credentials.
3. Perform steps 4 – 10 on Primary node first, then repeat 6 and 10 steps only on all the remaining replica/data node.
4. SSH to the Primary node as root user.
5. Delete the stale ops-lcm Service Registry entry via the CaSA REST API:
   

   curl --request DELETE --url https://localhost/casa/services/ops-lcm \
   
     --header 'accept: application/json' \
   
     --header 'content-type: application/json' \
   
     -u 'admin:PASSWORD' -k -i

6. Remove the CaSA B2B library cache:
   

   rm -fr /usr/lib/vmware-casa-b2b/

7. Change to the pak repository directory:
   

   cd /storage/db/pakRepoLocal/

8. Move aside the stale pak directories associated with the failed/retried upgrade attempt so they aren't picked up again on retry (adjust the pattern to match the actual pak/version directories relevant to your case):
   

   mv 9.1.1.0* 9.1.1.*-moved
   
   mv APUAT-for-8186-9110* APUAT-for-8186-9110*-moved

9. Edit vcopsPakManagerCommonHistory.json and remove all history entries referencing the paks moved in step 8 (matching "pak": "APUAT-for-8186-9110..." entries). Keep valid JSON — the last remaining object in the array must not have a trailing comma after its closing `}`.
   
   
10. Restart the CaSA service so it reloads its now-consistent state.
    

    service vmware-casa restart

11. Resume the upgrade from the Fleet Lifecycle page, re-run the "Run Component Upgrade Prechecks" task.