ESXi Network Adapter Shows Repeated "IO was aborted" Errors in VMkernel Logs
search cancel

ESXi Network Adapter Shows Repeated "IO was aborted" Errors in VMkernel Logs

book

Article ID: 381949

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • ESXi hosts may show repeated "command failed: IO was aborted" errors in VMkernel logs related to network adapter operations
  • These errors often occur hundreds of times per hour.
  • Multiple instances of QueryNicVportContext or similar network adapter query failures
  • Consistent error pattern over extended periods
  • Example vRealize Log Insight related to an ESXi host running a Mellanox network adapter:
    • "nmlx5_QueryNicVportContext:188 command failed: IO was aborted"

Environment

6.x, 6.7, 7.x, 8.x, 8.0U3

Cause

  • This issue typically occurs when there is a mismatch between the network adapter's firmware version and the version recommended in the Broadcom Compatibility Guide (BCG).
  • The communication between ESXi and the network adapter may fail when the firmware is not at a compatible version.

Resolution

  1.  Identify your network adapter details:

    1. Log into ESXi host using SSH or ESXi Shell

    2. Identify the affected adapter:

      esxcli network nic list 

    3. Note the device name, vendor ID, and device ID

  2. Verify firmware and driver versions:

    1. View the list of network adapters using the driver mentioned in the error:

      esxcli network nic list

      esxcli network nic get -n vmnicX`

      Note: Replace # with your adapter number

  3. Obtain the vendor ID, device ID, sub-vendor ID, sub-device ID

    1. esxcfg-info -a |grep -A22 vmnic# |grep -B4 "Vendor Name"`

      Note: Replace # with your adapter number

  4. Check compatibility:

    1. Visit the Broadcom Compatibility Guide

    2. Select "IO Devices" under "Platform & Compute"

    3. Select your ESXi version

    4. Enter your adapter's vendor ID, device ID, sub-vendor ID, sub-device ID

    5. If you see no entries, set sub-device ID to "all"

    6. If you still see no entries, set sub-vendor ID to "all"

    7. Locate your specific device model in the results

      Note: Exclude the common prefix (0x) when entering the product IDs

  5. Update firmware if needed:

    1. Download the recommended firmware from your hardware vendor

    2. Follow vendor-specific firmware update procedures

  6. Verify version after update

    • esxcli network nic get -n vmnic#

      Note: Replace # with your adapter number

  7. Validate resolution:

    1. Monitor vmkernel.log for at least 24 hours

    2. Verify error messages no longer appear

    3. If using Ariua for Logs, review for an absence of related alerts

Additional Information

  • Always check hardware vendor documentation for specific firmware update procedures
  • Consider scheduling firmware updates during maintenance windows
  • Back up host configuration before performing updates
  • Some vendors may require specific tools or methods for firmware updates

Performance impact of these errors varies by vendor. While some adapters may continue functioning with minimal impact, others may experience degraded performance or increased latency. The frequency of errors (often 150+ per hour) suggests this should be treated as a priority issue despite potentially minimal visible impact.

When working with customers:

Powered by Wolken Software