VCF Operations upgrade from 9.0.1 to 9.0.2 fails with error LCMVROPSYSTEM25008

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• An upgrade of VCF Operations (Aria Operations) from version 9.0.1 to 9.0.2 fails during the Product PAK upload/validation phase. This prevents the environment from reaching the target version.
• In the Fleet Management UI, you will notice an error code 'LCMVROPSYSTEM25008'.

• Under Fleet Management /var/log/vrlcm/vmware_vrlcm.log, you will notice the upgrade failure errors as shown below:

YYYY-MM-DDT04:18:31.835Z ERROR vrlcm[1219] [pool-3-thread-85] [c.v.v.l.p.c.v.t.InstallvROpsProductUpgradePakTask]  -- VMware Aria Operations product (application) upgrade failed
YYYY-MM-DDT04:18:31.835Z INFO vrlcm[1219] [pool-3-thread-85] [c.v.v.l.p.a.s.Task]  -- Injecting task failure event. Error Code : 'LCMVROPSYSTEM25008', Retry : 'true', Causing Properties : '{ CAUSE :: VAFileName ===  }'
com.vmware.vrealize.lcm.plugin.common.vrops.exceptions.UpgradeException: {"error_message_key":"general.failure","error_arguments":[],"error_message":"Operation failed"}
        at com.vmware.vrealize.lcm.plugin.core.vrops.tasks.InstallvROpsProductUpgradePakTask.execute(InstallvROpsProductUpgradePakTask.java:162) [vmlcm-vropsplugin-core-9.0.2.0-SNAPSHOT.jar!/:?]
        at com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:62) [vmlcm-engineservice-core-9.0.2.0-SNAPSHOT.jar!/:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.base/java.lang.Thread.run(Unknown Source) [?:?]

• The Software Update page is empty under VCF Operations Admin page:

Environment

VCF Operations 9.x

Cause

The directory /usr/lib/vmware-casa-b2b/casa-b2b remains orphaned on the appliance after a previous failed upgrade, a failed Aria Operations Pre-Upgrade Readiness Assessment Tool (APUAT) run, or incomplete cleanup logic in the PAK manager. The presence of this directory causes the subsequent upgrade process to follow an incorrect code path, resulting in the LCMVROPSYSTEM25008 failure.

In the VCF Operations node, from /storage/log/vcops/log/casa/casa.log you may see the following message:

YYYY-MM-DDT04:00:45.370Z WARN  casa 1479 [ops@4413 threadId="179" threadName="ajp-nio-127.0.0.1-8011-exec-12" requestId="########"] [PAK_MANAGER_QUERY_LOGGER.translateCommandLineException:590] - Unable to parse resource result: com.vmware.vcops.casa.exception.CasaException: JSON error unmarshalling; class=com.vmware.vcops.casa.upgrade.pak.PakManDetail, json=

YYYY-MM-DDT04:00:45.371Z ERROR casa 1479 [ops@4413 threadId="179" threadName="ajp-nio-127.0.0.1-8011-exec-12" requestId="########"] [PAK_MANAGER_ACTION_LOGGER.uploadToMaster:4909] - 

com.vmware.vcops.casa.exception.CasaException: Could not query_pak_signature for pakID=/storage/db/casa/pak/dist_pak_files/cargo/reservedTemp_11169171789368282275.pak

at com.vmware.vcops.casa.upgrade.pak.PakCommand.translateCommandLineException(PakCommand.java:591) ~[classes/:?]

Resolution

Note: This issue is officially resolved in VCF Operations version 9.1, where additional cleanup logic ensures the directory is removed automatically.

To workaround the issue and allow the upgrade to proceed, manually clean up the orphaned directory on all nodes in the cluster:

1. Stop the CASA service on all nodes in the following order:

   • All Data Nodes

   • The Replica Node (if present)

   • The Master Node

   • Command: service vmware-casa stop

2. Remove the orphaned directory on all nodes:

   • Check for the existence of /usr/lib/vmware-casa-b2b/casa-b2b/

   • If it exists, delete it: rm -rf /usr/lib/vmware-casa-b2b/casa-b2b/

3. Start the CASA service in the reverse order:

   • The Master Node

   • The Replica Node

   • All Data Nodes

   • Command: service vmware-casa start

4. Delete the failed upgrade task in Fleet Management and re-trigger the upgrade.