During VCF Operations upgrade from 9.0 to 9.0.1, Fleet manager appliance fails at Shut Down Guest OS phase (Stage 2). ERROR: LCMVSPHERECONFIG1000081

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

VCF Operations upgrade fails after the VCF Operations cluster is taken offline. Fleet Manager fails to find the VMs in the vCenter inventory, even thought they are present. There is reference to a folder named TEST in the Fleet Manager inventory, even though no such folder exists in vCenter:

because of this faulty view of the inventory, Fleet Manager is unable to find the VM. In the /var/log/vrlcm/vmware_vrlcm.log file we see entries like below.

INFO vrlcm[2152476] [pool-3-thread-80] [c.v.v.l.d.v.v.u.CoreUtility]  -- VM search name : <vm-name> and Cluster is :<cluster-name> and Folder is :TEST
INFO vrlcm[2152476] [pool-3-thread-82] [c.v.v.l.d.v.v.u.CoreUtility]  -- VM search name : <vm-name> and Cluster is :<cluster-name> and Folder is :TEST
INFO vrlcm[2152476] [pool-3-thread-80] [c.v.v.l.d.v.v.u.CoreUtility]  -- VM Name  : <vm-name> NOT found for cluster: <cluster-name>
INFO vrlcm[2152476] [pool-3-thread-80] [c.v.v.l.d.v.v.g.VsphereGuestOSOperation]  -- VirtualMachine not found or vm doesn't have vmware tools running
INFO vrlcm[2152476] [pool-3-thread-80] [c.v.v.l.p.a.s.Task]  -- Injecting task failure event. Error Code : 'LCMVSPHERECONFIG1000081', Retry : 'true', Causing Properties : '{ CAUSE ::  }' 
java.lang.Exception: Failed to shutdown VM. Check logs for additional details.
	at com.vmware.vrealize.lcm.plugin.core.vsphere.tasks.ShutdownGuestOSTask.execute(ShutdownGuestOSTask.java:106) [vmlcm-vsphereplugin-core-9.0.1.0-SNAPSHOT.jar!/:?]
	at com.vmware.vrealize.lcm.platform.automata.service.Task.retry(Task.java:158) [vmlcm-engineservice-core-9.0.1.0-SNAPSHOT.jar!/:?]
	at com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:60) [vmlcm-engineservice-core-9.0.1.0-SNAPSHOT.jar!/:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.base/java.lang.Thread.run(Unknown Source) [?:?]
INFO vrlcm[2152476] [pool-3-thread-82] [c.v.v.l.d.v.v.u.CoreUtility]  -- VM Name  : <vm-name> NOT found for cluster: <cluster-name>

Environment

VCF Operations 9.0

Cause

Fleet manager is looking for VM in incorrect location 

Resolution

Perform below steps :

1. 1. Go to the 'Failed' Upgrade task in Fleet manager → Lifecycle → VCF Management → Tasks page . Copy the Request ID from the URL.
   2. Access the swagger UI for Fleet management using How to Authorize VCF Operations Fleet Management API KB article.
   3. Cancel the Failed request using API call in below screenshot.
   4. 
   5. Do an 'Sync Inventory' for the management domain form VCF Operations → Inventory → VCF Instances → Management Domain →Actions 
      
   6. Try the upgrade again from Fleet manager → Lifecycle → VCF Management → Plan Upgrade page.