A subset of ESXI hosts leveraging NVMe over TCP devices encounter intermittent but frequent path loss
search cancel

A subset of ESXI hosts leveraging NVMe over TCP devices encounter intermittent but frequent path loss

book

Article ID: 407943

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Some, but not all, of the ESXi hosts in the cluster connected to VMFS datastores backed by NVMe over TCP storage devices may have frequent path loss events occur.

"Resetting controller" messages along with dead path(s) messages may be intermittently seen in vmkernel.log:

  • 2025-08-12T00:26:51.653Z In(182) vmkernel: cpu50:2467849)NVMEDEV:8245 Resetting controller 266 (nqn.X:Y.Z#vmhba65#.#.#.#:4420)
    ...
    2025-08-12T00:26:55.184Z In(182) vmkernel: cpu50:2097840)HPP: HppNvmeUpdateNamespaces:535: Marking paths dead - controller:nqn.X:Y.Z#vmhba65##.#.#.#:4420
    2025-08-12T00:26:55.184Z In(182) vmkernel: cpu50:2097840)HPP: HppPathGroupMovePath:688: Path "vmhba65:C0:T2:L54" state changed from "active" to "dead"
    2025-08-12T00:26:55.185Z In(182) vmkernel: cpu50:2097840)HPP: HppNvmeUpdateNamespaces:535: Marking paths dead - controller:nqn.X.Y:flasharray.Z#vmhba65##.#.#.#:4420

vobd.log will report the path(s) as degraded and down as well:

  • 2025-08-12T00:26:55.187Z In(14) vobd[2097958]:  [psastorCorrelator] 3684877445497us: [esx.problem.storage.redundancy.degraded] Path redundancy to storage device eui.## degraded. Path vmhba65:C0:T2:L54 is down. Affected datastores: "DATASTORE_NAME".

Environment

ESXi

NVMe over TCP storage configuration

Cause

A NVMe over TCP adapter binds to vmkernel interface(s) for connectivity to storage.  If there is a duplicate vmkernel IP assigned between ESXi hosts and bound to their respective NVMe over TCP adapter, this will introduce instability into the NVMe over TCP storage connections/sessions.

Resolution

Review IP addresses across the ESXi cluster to ensure there are no duplicate IP addresses in use in respect to vmkernel interfaces bound to the NVMe over TCP adapter.

Tools that can be used:

  • UI
    • vCenter, Host, Configure, Storage, Storage Adapters
    • vCenter, Host, Configure, Networking, Virtual Switches
    • vCenter, Host, Configure, Networking, VMkernel adapters
  • CLI
    • esxcfg-vswitch -l
    • esxcfg-vmknic -l
    • esxcli nvme adapter list
Powered by Wolken Software