NSX Controller Channel Down due to ESXi Storage Latency.
search cancel

NSX Controller Channel Down due to ESXi Storage Latency.

book

Article ID: 449088

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Intermittent transport node connectivity alarms occur across the ESXi cluster. Symptoms include:

  • NSX Manager reports the alarm: Controller Channel To Transport Node Down or Management Channel To Transport Node Down.
  • The alarm auto-resolves within minutes but recurs frequently.
  • ESXi vmkernel.log reveals storage performance deterioration: WARNING: ScsiDeviceIO: 1780: Device naa.#### performance has deteriorated. I/O latency increased from average value of #### microseconds to #### microseconds.
  • SCSI command failures with sense code H:0x0 D:0x28 P:0x0 appear in logs.
  • From the vmkernel.log
  2026-07-04T11:46:33.597Z In(182) vmkernel: cpu90:2098366)ScsiDeviceIO: 4633: Cmd(0x45bad3188540) 0x8a, CmdSN 0x800e0037 from world 2110622 to dev "[NAA_ID_1]" failed H:0x0 D:0x28 P:0x0
  2026-07-04T11:46:33.773Z In(182) vmkernel: cpu50:2098366)ScsiDeviceIO: 4633: Cmd(0x45dadb28e340) 0x2a, CmdSN 0x312 from world 2112286 to dev "[NAA_ID_1]" failed H:0x0 D:0x28 P:0x0
  2026-07-04T11:48:26.337Z In(182) vmkernel: cpu48:2098366)NMP: nmp_ThrottleLogForDevice:3898: H:0x0 D:0x28 P:0x0 . Act:NONE. cmdId.initiator=0x430fe5b0d480 CmdSN 0x800e004b

 

Environment

  • VMware NSX 
  • VMware vSphere ESXi 7.x/8.x
  • Fibre Channel (FC) or iSCSI Storage Arrays

Cause

The ESXi host experiences array-side resource exhaustion or fabric congestion, manifesting as a TASK_SET_FULL condition (SCSI status D:0x28). Because the local NSX configuration database (nsx-nestdb) requires continuous disk access, severe storage latency causes the database service to hang. When nsx-nestdb is unresponsive, the host cannot send management heartbeats to the NSX Manager, triggering the connectivity alarm.

If host is actively receiving H:0x0 D:0x28 P:0x0 SCSI sense codes across multiple devices. The D:0x28 status translates to a TASK_SET_FULL condition, which is returned when the storage array explicitly rejects incoming SCSI commands from the initiator because the array-side queue depth is completely exhausted.

Resolution

  1. Identify affected datastores by correlating NAA IDs in vmkernel.log warnings with the environment using esxcli storage vmfs extent list.
  2. Monitor storage performance using esxtop. Press u for the disk device view and check DAVG/cmd. Values consistently above 10-20ms indicate external storage or fabric issues.
  3. Review the physical storage array for Queue Full conditions, controller CPU saturation, or cache exhaustion.
  4. Inspect the SAN fabric (Fibre Channel switches) for port errors, CRC errors, or dropped frames that could delay SCSI command completion.
  5. Engage the storage hardware vendor to analyze array-side performance metrics during the incident window.
  6. Update to the latest patch release for your version of ESXi and NSX to ensure optimal driver and service stability. See Download Broadcom Products and Software to download these releases.
Powered by Wolken Software