NSX Host transport nodes down after storage outage or nestdb service failure

Products

• VMware NSX

Issue/Introduction

Following a storage outage or datastore disconnection, NSX transport nodes may appear "Down" in the NSX UI. This state leads to the failure of NSX-managed services, including DHCP server assignments, vMotion operations, and TKGI (Kubernetes) cluster node connectivity.

• ESXi hosts show as "Down" or "Unresponsive" in the NSX Manager UI.
• nsx-nestdb service is not running. You can verify the nsx-nestdb service status on the ESXi host via SSH: /etc/init.d/nsx-nestdb status
• vMotion operations fail for VMs on impacted hosts.
• Virtual Machines lose network connectivity or fail to receive DHCP IP addresses.
• TKGI/Kubernetes clusters report nodes in an "Unhealthy" or "Unresponsive" state.
• Guest operating systems may remount filesystems as "read-only."

Environment

VMware NSX

VMware Tanzu Kubernetes Grid Integrated Edition (TKGI)

VMware vSphere ESXi

Cause

A storage outage or scratch datastore disconnection prevents the nsx-nestdb service from starting or staying running. Even after storage recovery, residual IO references (APD state) can prevent the service from initializing correctly.

Resolution

1. Verify storage connectivity to the impacted ESXi hosts in vCenter.
2. Perform a full reboot of the ESXi hosts showing as "Down" in the NSX UI to clear APD states and restart NSX agents.
3. For TKGI environments, if nodes remain unresponsive after the host reboot, perform a bosh recreate on the impacted VMs to restore "read-only" filesystems to a "read-write" state.
4. Verify that the nsx-nestdb service is running on the host via SSH: /etc/init.d/nsx-nestdb status

If the issue persists, refer to Contact Support for further assistance.

Additional Information

Permanent Device Loss (PDL) and All-Paths-Down (APD) on host