Esxi host rebooted and unable to find the operating system

Products

• VMware vSAN

Issue/Introduction

A vSAN host is not longer able to boot to ESXi and is unable to find the ESXi OS 

When booting to an ISO or ESXi installer we can see no ESXi is installed but disks are claimed by vSAN 

Environment

vSAN (all) 

Cause

A number of actions or event can cause this including but not limited to hardware issues

Resolution

Engage with hardware vendor to review. 

If ESXi is needed to be reinstalled 

To rejoin the ESXi host to the vSAN cluster:

1. Remove the old ESXi host entry from vCenter 
   1. Validate the ESXi is in a disconnected state. If so proceed to step 3. 
   2. If it is connected Right-click the host you want to Disconnect in the inventory pane, and select disconnect from the pop-up menu.
   3. Right click on the ESX host you wish to remove from the Inventory and select "remove from inventory"
      
      
2. Install the exact ESXi version (Via build number) as the other remaining hosts in the cluster, ensuring that you preserve the vSAN disk partitions. Build numbers and versions of VMware ESXi/ESX
3. Add the host to the Datacenter Object in vCenter Server. Creating a datacenter and adding an ESXi host to the vCenter Server Inventory using vCenter Server appliance.
4. Configure the vSAN VMkernel port group on the host. For more information, see How to configure vSAN VMkernel networking.
5. Drag the host in to the vSAN cluster object. This will enable vSAN clustering on the host and trigger a vSAN cluster update pushing the correct unicastagent list to all hosts.
6. Validate cluster health via the vSAN skyline health check 

If the process above does not correct the issue please follow the steps below. 

1. Connect to one of the remaining vSAN cluster hosts using SSH.
2. Identify the vSAN Sub Cluster ID using this command:
   
   

   # esxcli vsan cluster get

   
   
   You see output similar to:
   
   

   Cluster Information
   Enabled: true
   Current Local Time: YYYY-MM-DDTHH:MM:SS
   Local Node UUID: ########-####-####-####-########826f
   Local Node Type: NORMAL
   Local Node State: AGENT
   Local Node Health State: HEALTHY
   Sub-Cluster Master UUID: ########-####-####-####-########f17d
   Sub-Cluster Backup UUID: ########-####-####-####-########dd93
   Sub-Cluster UUID: ########-####-####-####-########9e45
   Sub-Cluster Membership Entry Revision: 2
   Sub-Cluster Member Count: 3
   Sub-Cluster Member UUIDs: ########-####-####-####-########f17d, ########-####-####-####-########dd93, ########-####-####-####-########826f
   Sub-Cluster Member HostNames: esxi3.########, esxi2.########, esxi1.########
   Sub-Cluster Membership UUID: ########-####-####-####-########f17d
   Unicast Mode Enabled: true
   Maintenance Mode State: OFF
   Config Generation: ########-####-####-####-########d2c2 3 YYYY-MM-DDTHH:MM:SS
   Mode: REGULAR
   vSAN ESA Enabled: false

    
3. Run one of the commands below on the newly rebuilt ESXi host using the Sub Cluster UUID identified in step 2:
   • For vSAN OSA:
     
     

     # esxcli vsan cluster join -u sub_cluster_UUID

     
     
     For example:
     
     

     # esxcli vsan cluster join -u ########-####-####-####-########9e45

     
     
     
   • For vSAN ESA:
     

     # esxcli vsan cluster join -x -u sub_cluster_UUID

     
     For example:
     
     

     # esxcli vsan cluster join -x -u ########-####-####-####-########9e45

     
     
     
      
4. Verify that the host is now a part of the vSAN cluster by running the command:
   
   

   # esxcli vsan cluster get

   
   You see output similar to:
   
   

   Cluster Information
   Enabled: true
   Current Local Time: YYYY-MM-DDTHH:MM:SS
   Local Node UUID: ########-####-####-####-########965e
   Local Node Type: NORMAL
   Local Node State: AGENT
   Local Node Health State: HEALTHY
   Sub-Cluster Master UUID: ########-####-####-####-########f17d
   Sub-Cluster Backup UUID: ########-####-####-####-########dd93
   Sub-Cluster UUID: ########-####-####-####-########9e45
   Sub-Cluster Membership Entry Revision: 3
   Sub-Cluster Member Count: 4
   Sub-Cluster Member UUIDs: ########-####-####-####-########f17d, ########-####-####-####-########dd93, ########-####-####-####-########826f, ########-####-####-####-########965e
   Sub-Cluster Member HostNames: esxi3.########, esxi2.########, esxi1.########, esxi4.########
   Sub-Cluster Membership UUID: ########-####-####-####-########f17d
   Unicast Mode Enabled: true
   Maintenance Mode State: OFF
   Config Generation: ########-####-####-####-########d2c2 4 YYYY-MM-DDTHH:MM:SS
   Mode: REGULAR
   vSAN ESA Enabled: false

    
5. In the vCenter Server, refresh the vSAN status view. All hosts now report the status as Healthy.

Additional Information

See KB How to add a host back to a vSAN cluster after an ESXi host rebuild/reimage for additional information