ESXi hosts experiences All Paths Down events on USB based SD Cards while using the vmkusb driver
Article ID: 323075
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
The purpose of this article is to highlight the tolerance level of the vmkusb driver to recover from brief SD card disconnections witnessed on the Hardware level.
Symptoms:
ESXi 6.x/7.x hosts experience random All Paths Down events (APDs) on their locally attached SD cards using the vmkusb based USB driver.
-
- Example log snippet from vobd.log:
YYYY-MM-DDTHH:MM:SS.###Z: [APDCorrelator] 1960326656825us: [vob.storage.apd.start] Device or filesystem with identifier [mpx.vmhba32:C0:T0:L0] has entered the All Paths Down state.YYYY-MM-DDTHH:MM:SS.###Z: [scsiCorrelator] 1960326656798us: [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba32:C0:T0:L0 changed state from onYYYY-MM-DDTHH:MM:SS.###Z: [APDCorrelator] 1960324330689us: [esx.problem.storage.apd.start] Device or filesystem with identifier [mpx.vmhba32:C0:T0:L0] has entered the All Paths Down state.YYYY-MM-DDTHH:MM:SS.###Z: [scsiCorrelator] 1960324334135us: [esx.problem.storage.connectivity.lost] Lost connectivity to storage device mpx.vmhba32:C0:T0:L0. Path vmhba32:C0:T0:L0 is down. Affected datastores: "".YYYY-MM-DDTHH:MM:SS.###Z: [netCorrelator] 1960335078949us: [vob.net.vmnic.linkstate.up] vmnic vusb0 linkstate upYYYY-MM-DDTHH:MM:SS.###Z: [netCorrelator] 1960332762933us: [esx.clear.net.vmnic.linkstate.up] Physical NIC vusb0 linkstate is upYYYY-MM-DDTHH:MM:SS.###Z: [APDCorrelator] 1960466657332us: [vob.storage.apd.timeout] Device or filesystem with identifier [mpx.vmhba32:C0:T0:L0] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will now be fast failed.
YYYY-MM-DDTHH:MM:SS.###Z: [APDCorrelator] 1960464330957us: [esx.problem.storage.apd.timeout] Device or filesystem with identifier [mpx.vmhba32:C0:T0:L0] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will now be fast failed. - Example log snippet from vmkernel.log:
YYYY-MM-DDTHH:MM:SS.###Z cpu0:2097360)WARNING: NMP: nmpUnclaimPath:1561: NMP device "mpx.vmhba32:C0:T0:L0" quiesce state change failed: Busy
YYYY-MM-DDTHH:MM:SS.###Z cpu0:2097360)WARNING: ScsiPath: 6506: Path vmhba32:C0:T0:L0 is being removed
YYYY-MM-DDTHH:MM:SS.###Z cpu0:2097360)WARNING: ScsiPath: 6820: Failed to issue command 0x0 (cmdSN 0x0) on path vmhba32:C0:T0:L0: No connection
- Example log snippet from vobd.log:
The ESXi host will remain connected inside vCenter, however, the ESXi operating system will be in a read only state until it is rebooted.
This issue does not re-occur if you move back to the legacy USB driver as outlined here: KB 344764.
*Note: The legacy USB/XHCI drivers are no longer available in ESXi 7.0 and beyond.
Environment
VMware ESXi 6.7.x
VMware ESXi 6.5.x
VMware vSphere ESXi 7.0.x
VMware ESXi 6.5.x
VMware vSphere ESXi 7.0.x
Cause
In ESXi 6.5, a new USB driver was introduced to replace the legacy linux based USB and XHCI drivers to be used for communication with SCSI based SD cards seen through onboard USB based controllers.
While the cause of the events may vary and depend on the Hardware layer, small disconnections from the SD card via the USB based interface while using the new vmkusb driver can result in an APD event that will most likely render the ESXi OS install location to drop. This causes the ESXi system to then go into read only mode, however, it can still run and operate under most normal conditions within vCenter and still service virtual machine operations.
Further, the ESXi OS may recover from the APD event once the SD card reconnects if the SD card is not configured to have an active core dump partition on it. However, this is not a suggested workaround.
This issue is not seen to re-occur inside the Linux based legacy USB driver. This is believed to be due to the maturity of the legacy code and that the resiliency on retrying loss connections to the USB device is more flexible, thus it recovers without any subsequent events or impact.
While the cause of the events may vary and depend on the Hardware layer, small disconnections from the SD card via the USB based interface while using the new vmkusb driver can result in an APD event that will most likely render the ESXi OS install location to drop. This causes the ESXi system to then go into read only mode, however, it can still run and operate under most normal conditions within vCenter and still service virtual machine operations.
Further, the ESXi OS may recover from the APD event once the SD card reconnects if the SD card is not configured to have an active core dump partition on it. However, this is not a suggested workaround.
This issue is not seen to re-occur inside the Linux based legacy USB driver. This is believed to be due to the maturity of the legacy code and that the resiliency on retrying loss connections to the USB device is more flexible, thus it recovers without any subsequent events or impact.
Resolution
VMware cannot identify and resolve the hardware layer disconnects being observed, however, VMware is aware of the resiliency behavior of the vmkusb driver and this is being addressed in a future vmkusb driver not yet released.
Please work with your server/SD card hardware vendor to address concerns of SD card disconnects.
For assistance with a workaround vmkusb driver, please open a support ticket and reference this KB article.
Please work with your server/SD card hardware vendor to address concerns of SD card disconnects.
For assistance with a workaround vmkusb driver, please open a support ticket and reference this KB article.
Additional Information
Impact/Risks:
As the SD cards are used as the install medium for the ESXi OS, the All Paths Down event can cause the ESXi OS to go into a read only state for all OS files and folders based on the SD card. A reboot is required to return the system to a read/write state. This issue does not impact virtual machines or the ESXi host capabilities in vCenter, such as SVMotion, VMotion, and HA events, etc.
As the SD cards are used as the install medium for the ESXi OS, the All Paths Down event can cause the ESXi OS to go into a read only state for all OS files and folders based on the SD card. A reboot is required to return the system to a read/write state. This issue does not impact virtual machines or the ESXi host capabilities in vCenter, such as SVMotion, VMotion, and HA events, etc.
Feedback
Yes
No
Powered by
