vApp/VMs are in inconsistent state on VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

VMs / vApps are in inconsistent state in VMware Cloud Director, however they are in powered on state in vCenter.

Environment

VMware Cloud Director 10.x

Cause

This issue can occur when there are discrepancies between Power States in vCenter and Cloud Director.

• The most common scenario is manual power change of VCD VMs in vCenter.
• Other scenarios such as a HA Event may be responsible for the discrepancies.

In any scenario, VCD would generally receive an event to state the Power State now differs and will try to align.

• Once VCD begins this alignment, it is not expecting the state to vary i.e. if the VM was off, it expects it to remain off until vCenter processes it's ask to Power On.
• Should vCenter Power On/Off outside of the VCD ask, it may leave the VM in a state where the VCD ask now contradicts the state.
  • VCD is asking for the VM to be Powered On, in VC it is already on.
• If the VCD Alignment task fails, it will revert to it's previous state, which now contradicts the VC state, leading to an Inconsistent state in Cloud Director.

 

Resolution

To resolve this issue:

1. Find a VM in <Inconsistent> state.
2. Under General tab for the affected VM note the <Name in vSphere>.
3. Log in to the vCenter Server.
4. Find the VM with the name found in step 2.
5. Right-click on the VM in vCenter Server and click Power > Shutdown Guest OS or Power Off.
6. In the VCD UI, on the affected VM Click on All actions > Power >  Power On.

 

In Cloud Director 10.6, a troubleshooting mechanism was added to the Cell Management Tool to identify and help resolve situations where this inconsistency occurs.

• Currently there is no code to account for a mid-flight state change, so the inconsistency will still occur.
• Engineering are actively aware of this and are looking to scope a fix for inclusion in a future release.

The troubleshooting mechanism is currently internal and therefore not externally documented, please engage Broadcom Support for assistance.