ESXi Network Adapter Shows Repeated "IO was aborted" Errors in VMkernel Logs

Products

• VMware vSphere ESXi

Issue/Introduction

• ESXi hosts may show repeated "command failed: IO was aborted" errors in VMkernel logs related to network adapter operations
• These errors often occur hundreds of times per hour.
• Multiple instances of QueryNicVportContext or similar network adapter query failures
• Consistent error pattern over extended periods
• Example vRealize Log Insight related to an ESXi host running a Mellanox network adapter:
  • "nmlx5_QueryNicVportContext:188 command failed: IO was aborted"

Environment

- VMware vSphere 6.7 and later
- Network adapters with firmware-managed capabilities

Cause

This issue typically occurs when there is a mismatch between the network adapter's firmware version and the version recommended in the Broadcom Compatibility Guide (BCG). The communication between ESXi and the network adapter may fail when the firmware is not at a compatible version.

Resolution

1.  Identify your network adapter details:
   1. Log into ESXi host using SSH or ESXi Shell
   2. Run `esxcli network nic list` to identify the affected adapter
   3. Note the device name, vendor ID, and device ID
      
      
2. Verify firmware and driver versions:
   1. Run `esxcli network nic list` to see the list of network adapters using the driver mentioned in the error
   2. Run `esxcli network nic get -n vmnicX` (replace X with your adapter number)
      
      
3. Obtain the vendor ID, device ID, sub-vendor ID, sub-device ID
   1. run `esxcfg-info -a |grep -A22 vmnicX |grep -B4 "Vendor Name"` (replace X with your adapter number)
      
      
4. Check compatibility:
   1. Visit the Broadcom Compatibility Guide at https://compatibilityguide.broadcom.com/
   2. Select "IO Devices" under "Platform & Compute"
   3. Select your ESXi version
   4. Enter your adapter's vendor ID, device ID, sub-vendor ID, sub-device ID
   5. If you see no entries, set sub-device ID to "all"
   6. If you still see no entries, set sub-vendor ID to "all"
   7. Locate your specific device model in the results
      
      
5. 4. Update firmware if needed:
   1. Download the recommended firmware from your hardware vendor
   2. Follow vendor-specific firmware update procedures
      
      
6. Verify version after update
   • Run `esxcli network nic get -n vmnicX` (replace X with your adapter number)
     
     
7. Validate resolution:
   1. Monitor vmkernel.log for at least 24 hours
   2. Verify error messages no longer appear
   3. If using vRealize Log Insight, check vRLI for absence of related alerts

Additional Information

• Always check hardware vendor documentation for specific firmware update procedures
• Consider scheduling firmware updates during maintenance windows
• Back up host configuration before performing updates
• Some vendors may require specific tools or methods for firmware updates

Performance impact of these errors varies by vendor. While some adapters may continue functioning with minimal impact, others may experience degraded performance or increased latency. The frequency of errors (often 150+ per hour) suggests this should be treated as a priority issue despite potentially minimal visible impact.

When working with customers:
1. Consider maintenance window requirements
2. Document pre-update configurations
3. Be prepared to assist with vendor escalation if needed
4. Note that certain vendors may require specific update tools not covered in this KB


Determining Network/Storage firmware and driver version in ESXi