Power-Off VCF Automation Nodes Fails in Fleet Management with Error LCMVMSP10028

Products

• VCF Automation

Issue/Introduction

• When attempting to power off VCF Automation nodes from Fleet Management within VCF Operations (via VCF Operations UI > Fleet Management > Lifecycle > VCF Management > Components > Automation > Manage > Power off), the task fails.
• Reviewing the task failure logs in /var/log/vrlcm/vrlcm.log on the Fleet Management Appliance reveals the following error trace:
  
  YYYY-MM-DDT00:13:45.857Z INFO vrlcm[1640695] [pool-3-thread-14] [c.v.v.l.p.a.s.Task]  -- Injecting task failure event. Error Code : 'LCMVMSP10028', Retry : 'true', Causing Properties : '{ CAUSE ::  }'
java.lang.RuntimeException: Unable to power-off VIDB nodes in the vCenter.You may try to manually power off the VMs: [Node{vmName='vcfa-node1'}, Node{vmName='vcfa-node2'}, Node{vmName='vcfa-node3'}] in the vCenter
        at com.vmware.vrealize.lcm.vmsp.plugin.tasks.powerops.ClusterVspherePowerOffTask.execute(ClusterVspherePowerOffTask.java:79)
  
  Note: Although the logs indicate VIDB nodes, the intended targets are actually the Automation VMs.

Environment

• VMware VCF Operations 9.0.x (Fleet Management)

• VMware VCF Automation 9.0.x

 

Cause

The ClusterVspherePowerOffTask execution failed because vCenter Server did not complete or acknowledge the power-down API request for the target virtual machines within the designated timeout period due to unresponsive VMware Tools inside the guest operating system or stale task locks in vCenter Server.

Resolution

1. Log in to the managing vCenter Server interface using administrative credentials.

2. Search for the impacted VCF Automation node virtual machines:

   • vcfa-node1

   • vcfa-node2

   • vcfa-node3

3. Review the current power state and active task history for each virtual machine.

4. Initiate a Guest OS Shutdown command via the vSphere Client.

5. If the Guest OS does not respond within 3 minutes, issue a forced Power Off command.

6. Power on the target virtual machines again.

7. Once all target virtual machines are confirmed in a powered-on state in vCenter Server, navigate back to Fleet Manager in VCF Operations (Fleet Management > Lifecycle > VCF Management > Components > Automation).

8. Trigger an Inventory sync to reconcile the state between Fleet Management and vCenter Server.