VC/hostd/vmx files are out of sync in vSAN

Products

• VMware vSAN

Issue/Introduction

Identify and resolve out of sync VM's in vSAN.

Symptoms:

• VMs are being detected out of sync when using vsan.check_state in RVC.
• In vsan.check_state log file, you see entries similar to,

2018-05-29 14:41:48 +0000: Step 1: Check for inaccessible vSAN objects
Detected 0 objects to be inaccessible
2018-05-29 14:41:49 +0000: Step 2: Check for invalid/inaccessible VMs
2018-05-29 14:41:49 +0000: Step 3: Check for VMs for which VC/hostd/vmx are out of sync
Find the list of VMs for which VC/hostd/vmx are out of synced.

Environment

VMware vSAN 6.6.x

Cause

If VM has migrated from one ESXi host to another, there can be a leftover/ghost process on the original ESXi host, which can lead to the out of sync status in RVC.

Resolution

1. For each VM in the list of out of sync VM's from the output of the vsan.check_state command, check in vCenter which ESXi host the VM is actually running on.
2. On each ESXi host in the vSAN cluster, use the LSOF (list open files) command to search for running processes related to the VM.

# lsof | grep "VM name"

"Process ID"       "Process name" 

1. If a running process is found for the VM, even though the VM is actually running on a different ESXi host, this is the cause of the out of sync state. Kill this ghost process running the command,

kill -9 "Process ID"

1. Re-run vsan.check_state, and confirm the VM is no longer out of sync.

Note: Ensure the process being killed is the ghost process, and not the actual running process for the VM on the ESXi host the VM is running on.

Additional Information

ESXi hosts showing out of sync with distributed switch

Impact/Risks:
If killing a process, ensure the command is run on the correct ESXi host, so as not to affect any running VM.