vSAN -- File Services -- File Service VM (= FSVM) reports: "Virtual Machine Consolidation Needed Status"

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
This issue affects the automated generated File Service VMs (= FSVM).
Reference:  
"When you configure vSAN file service, vSAN creates a single VDFS distributed file system for the cluster which will be used internally for management purposes. 
A file service VM (FSVM) is placed on each host. The FSVMs manage file shares in the vSAN datastore. Each FSVM contains a file server that provides both NFS and SMB service."
 

• A File Service VM (= FSVM) shows the Warning: "Virtual Machine Consolidation Needed Status"
• The vCenter /var/log/vmware/vsan-health/vmware-vsan-health-service.log indicate that the destroy of the File Service VM (= FSVM) failed by showing similar entries like the following:

2020-04-21T06:05:20.787Z ERROR vsan-mgmt[35201] [VsanEamUtil::DestroyEamAgentVMs opID=W79787] Failed to destroy agent VM 'vim.VirtualMachine:vm-333' on host host-108
Traceback (most recent call last):
  File "/build/mts/release/bora-16051639/bora/build/vcenter/release/pkg/vsanmgmt/stage/usr/lib/vmware-vpx/vsan-health/pyMoVsan/VsanEamUtil.py", line 734, in DestroyEamAgentVMs
  File "/usr/lib/vmware/site-packages/pyVim/vimutil.py", line 30, in InvokeAndTrack
    return WaitForTask(task)
  File "/usr/lib/vmware/site-packages/pyVim/task.py", line 140, in WaitForTask
    raise info.error
VsanVmomiPatcher.vim.fault.InvalidPowerState: (vim.fault.InvalidPowerState) {
   msg = 'The attempted operation cannot be performed in the current state (Powered on).',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   requestedState = 'poweredOff',
   existingState = 'poweredOn'
}
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere 7.0.x

Cause

This issue occurs when the File Service Remediation Service detected one or more File Service VM (= FSVM) is not working 
and tries to remove and deploy a new File Service VM (= FSVM).
In the process of removing the bad File Service VM (= FSVM), the host side File Service Recovery Service might power it back. This will make the File Service VM (= FSVM) to be partially removed.

Resolution

This issue has been resolved by upgrading vCenter/ESXi to version 7.0 Update 1.


Workaround:

Workaround to resolve this issue:

1.) Power off the File Service VM (= FSVM) that reported the Warning: "Virtual Machine Consolidation Needed Status"
2.) Right-click the VM and select "Delete from Disk" to remove the VM completely. 
3.) The File Service Remediation process will deploy a new File Service VM (= FSVM) onto the same Host after Step 2 has been completed by the System.