VCF Operations for Logs upgrade via Fleet Management fails at stage 8 with LCMVRLICONFIG40040

Products

• VCF Operations

Issue/Introduction

The upgrade of VCF Operations for Logs from version 9.0.1 to 9.0.2 initiated through VCF Operations – Fleet Management failed at Stage 8 with error LCMVRLICONFIG40040.
The failure occurred during the stage “Start Operation-logs Generic Task”, as confirmed by VCF Management tasks.

Although the Fleet Management workflow reports a failure, validation from the VCF Operations for Logs UI, nodetool-no-pass status | nodetool-no-pass describecluster, and cluster health checks confirmed that the upgrade had already completed successfully on all nodes.

vmware_vrlcm.log

 com.vmware.vrealize.lcm.common.exception.userinput.vrli.VrliUpgradeVersionNotSupportedException: Upgrade not needed as the version: 9.0.2.0.25137850. is same as current
 at com.vmware.vrealize.lcm.plugin.vrli.VrliUpgradeTask.execute(VrliUpgradeTask.java:195)
 at com.vmware.vrealize.lcm.platform.automata.service.Task.retry(Task.java:158)
 at com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:60)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.base/java.lang.Thread.run(Unknown Source)

Environment

VCF Operations for Logs 9.0.2
VCF Operations 9.0.2

Cause

The error LCMVRLICONFIG40040 would be triggered only when the RETRY option was selected, indicating that the target version (9.0.2.0.25137850) matched the currently installed version.
This behavior points to a race condition, where the upgrade completed on VCF Operations for Logs, but the corresponding update request to Fleet Management failed to reconcile the final state at Stage 8.

Resolution

1. Its important that we perform multiple validations (UI status, upgrade logs, Cassandra health, and node version checks) & confirm that the upgrade to 9.0.2 is completed successfully on VCF Operations for Logs.
   
   NOTE: Proceed with the below steps only if all the validations mentioned in steps is 1 is reporting green
   
   
2. The failed Fleet Management request at Stage 8 needs to be manually canceled using the Fleet Management API. Refer to this KB 415510
3. After canceling the request, trigger an inventory sync and wait for it complete & the status will be updated to is compliance in Fleet Management.

Additional Information

How to Authorize VCF Operations Fleet Management API