ESXI Host went to maintenance mode and inaccessible state

Products

• VMware vSphere ESXi

Issue/Introduction

An ESXi host's status in vCenter Server changes to "Inaccessible," "Disconnected," or "Not Responding" after it was placed in or exited maintenance mode. This often occurs when network configurations were changed (either manually, via host profile, or an automated process) during or prior to the maintenance window, resulting in a loss of the host's management network connectivity.

Environment

VMware ESXi 8.x
VMware vCenter Server 8.x

Cause

The primary cause of an ESXi host becoming inaccessible is the loss of its management network connectivity. This can stem from various physical NIC (vmnic) or network misconfigurations, often introduced during maintenance operations:

Physical Switch Port Misconfiguration : 

Incorrect VLAN Tagging: The switch port connected to the ESXi host's management vmnic is configured with the wrong VLAN ID, or the necessary VLANs are not allowed on a trunk port.
Access vs. Trunk Mode: The switch port mode (e.g., access port for a single VLAN vs. trunk port for multiple VLANs) might be incorrect.
Cable Disconnection/Damage: A physical network cable connecting the ESXi host's vmnic to the physical switch is disconnected, damaged, or plugged into the wrong port.
Management vmkernal Adapter VLAN Mismatch: The VLAN ID configured on the ESXi host's management VMkernel adapter (vmk0 or similar) does not match the VLAN on the physical switch port.
Incorrect vmnic Assignment: The vmnic intended for management traffic was accidentally removed from the management vSwitch, or an incorrect vmnic was assigned.

Resolution

Few Steps that might help us diagnose for resolution as below : 

1. Ping the ESXi Host Management IP:
   • From a machine on the same management network, attempt to ping the ESXi host's management IP address. If it fails, network connectivity is indeed the issue.
2. Check Physical Network Status:
   • Visually inspect the physical network cables connected to the ESXi host. Ensure they are securely plugged into the correct vmnic ports on the host and the correct ports on the physical switch.
   • Check the link/activity lights on the vmnic ports on the ESXi host and the corresponding ports on the physical switch. They should be green/amber and blinking.
3. Access the ESXi Host's Direct Console User Interface (DCUI):
   • Connect a keyboard and monitor directly to the ESXi host. This is often the most reliable way to regain access when network connectivity is lost.
   • Press F2 to customize the system. You will need the root password.
4. Check Network Adapters:
   • • Select Network Adapters and press Enter.
     • Ensure that the correct vmnic (e.g., vmnic0) is selected and marked as active for the management network vSwitch.
     • Verify that the correct physical NICs are linked and showing "Up" status.
5. Check VLAN (Optional):
   • • Select VLAN (Optional) and press Enter.
     • Ensure the configured VLAN ID matches the VLAN ID on the physical switch port for the management network. If the management network is untagged, ensure this field is blank (0).
6. Test Management Network:
   Select Test Management Network and press Enter. This will perform basic ping tests to the default gateway, DNS server, and possibly vCenter

7. From Physical Switch CLI (If you suspect switch-side issue):

   1. Identify Uplink Ports: Determine which physical switch port(s) the ESXi host's vmnic(s) are connected to.
   2. Check Port Configuration:
      • Log in to the physical switch CLI.
      • Examine the configuration of the identified port(s).
      • Verify:
        • Port is no shutdown.
        • Port mode (e.g., switchport mode trunk or switchport mode access vlan X).
        • Allowed VLANs (if trunking: switchport trunk allowed vlan add X or switchport trunk allowed vlan Y,Z).
        • The VLAN ID for the ESXi management network is correctly configured and allowed.

8. Correct Physical Switch Port Configuration:

   • If the issue is on the physical switch, correct the port mode, allowed VLANs, or cable connections. Save the switch configuration.
     
     
9. • Restart Management Agents:
     bash /etc/init.d/hostd restart /etc/init.d/vpxa restart
     This can sometimes resolve issues where agents are simply hung.

Additional Information

Troubleshooting an ESXi host in a "not responding"/"disconnected" state

Restarting Management Agents in ESXi

Network connectivity is lost on ESXi host after reboot.

ESXi Host Is in Disconnected State and Unable to Connect to vCenter