vsan-healthcheck-disposable VMs fail to delete automatically in vSAN cluster.

Products

• VMware vSAN 7.x
• VMware vSAN 8.x

Issue/Introduction

Symptoms:

• By design, SDDC Manager runs vSAN Proactive test every few minutes.
  
  
• This vSAN Proactive Test has one of the test as VM Creation Test which creates VMs named 'vsan-healthcheck-disposable' as part of its testing.
  
  
  
  
  
• To navigate to 'VM Creation test : vSphere Client > Select vSAN Cluster > Under 'Monitor' tab > 'vSAN' > Proactive test > VM Creation Test
  
  
• Automatic deletion of these VMs does not work and are seen as stale VMs in vCenter inventory (in powered-off state) and these VM's are noticed as inaccessible on the vSAN skyline health check.
  
  

Environment

VMware vSAN 7.x
VMware vSAN 8.x (OSA and ESA)

Cause

• By default, these temporary virtual machines (vsan-healthcheck-disposable-####-######-esx-####) are deleted immediately after creation as part of the 'VM Creation Test.'
• During this test, one VM is created per host in the vSAN cluster.
• However, if the vSAN cluster experiences an active issue, performance disruption, or interrupted health check, these virtual machines may fail to clean up automatically and remain as stale objects in the vCenter inventory."

Resolution

• To verify the health of the vSAN cluster, run the Proactive Test > VM Creation Test.
  
  
• If the test completes successfully and the virtual machines are created and automatically deleted right away (as shown under Recent Tasks in the vSphere Client), the vSAN cluster is healthy and its underlying components are functioning normally. Automatic deletion of the VMs are indicative of good health of vSAN cluster.

• Because these are temporary health check objects, they contain no production data. Deleting these stale objects is safe, has no negative repercussions on vSAN functionality.

  Option A: vSphere Client (If registered as VMs)

  1. In the vSphere Web Client, locate the stale vsan-healthcheck-disposable VMs (typically in a powered-off state).

  2. Right-click the VM and select Delete from Disk.

  Option B: ESXi Shell (If objects are seen as unknown and Inaccessible)

  If the objects are seen as unknown and inaccessible and not visible in the vSphere Client inventory, they can be removed manually via the ESXi command line using their Object UUIDs.

  WARNING: Proceed with extreme caution. Object deletion via the CLI is permanent and cannot be undone. Always verify you are deleting the correct object UUID before executing the delete command.

  Steps:

  1. Connect to an ESXi host in the vSAN cluster via SSH as root.

  2. Validate the Object: Before deletion, verify the object's metadata and state by running.
     
     /usr/lib/vmware/osfs/bin/objtool getAttr -u <Object_UUID>

  3. Delete the Object: Once confirmed that the object is safe to remove, delete it using objtool.

/usr/lib/vmware/osfs/bin/objtool delete -u <Object_UUID>

  Note: If uncertain about the object status or need assistance, stop and open a case with Broadcom Support before proceeding.

Additional Information

• Subscribe for Updates: For tracking defect status and fixes, please subscribe to this article: Subscribe to a Broadcom knowledge article by article or product (KB 275360).
• Downloads: To access product patches, visit the Broadcom Product Downloads page.
• Support: To speak with a customer representative or a Support Engineer, see Contact Support. Scroll to the bottom of the page and click on your respective region.
• Retrieve Logs: For instructions on gathering logs, review the official VMware Documentation.