Out of inventory witness alerts seen in vSAN cluster Fault Domains

Products

• VMware vSAN

Issue/Introduction

Out of inventory witness alert via vSAN cluster > Configure > vSAN > Fault Domains. 

When trying to add host(s) to the cluster they're unable to join and will show as network partitioned

 

Environment

vSAN (All Versions)

 

Cause

• The Witness Appliance was not correctly added to the cluster
• Witness Appliance was removed from vCenter inventory and then re-added which changes the MOID.

 

 

Resolution

Follow the below workaround to remove the old witness entry from the vSAN MOB.

1. Place the witness appliance into maintenance mode

2. Open an SSH session to the vCenter appliance
3. Log into RVC with "rvc localhost" if still using [email protected] otherwise log in as rvc administrator@<domain>.local@localhost
4. Navigate to the vSAN stretch cluster with the non-working witness appliance

5. Enable the vSAN MOB, disabled by default, by running the command vsan.debug.mob --start /localhost

6. Access the URL link provided and log in with [email protected] or what ever you changed the domain to and password

7. Click on (more...) then select vsan-stretched-cluster-system

8. Select VSANVcRemoveWitnessHost option in the vSAN MOB.

9. Select the vSAN cluster having the issue in vCenter. Copy the cluster domain ID from the URL of the browser. It should be similar to 'domain-c<number>', not the entire string.

 

10. From vCenter > vSAN Cluster > Configure > vSAN > Fault Domains click on the "Out of Inventory witness" to get the host ID of the witness appliance from the URL.

You can also get the host ID via RVC as well

11. Enter the <domain-c> and <host-> values which we copied from the URL into their respected sections in VSANVCRemoveWitnesHost option as seen in the below screenshot.

12. Reconfigure the stretch cluster

13. Once the stretch cluster is reconfigured the partitioned host will now be part of the cluster or you can add additional hosts successfully to the cluster.