On any HPE server running VMware ESXi 6.0 U3, VMware ESXi 6.5 or VMware ESXi 6.7 and configured with any of the HPE Broadcom-Based network adapters with bnxtnet driver version 212.0.119.0 (or prior) may experience a PSOD.

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• ESXi hosts running on HPE Custom Image 6.0 U3; 6.5 and 6.7 with below network card models and configured with any of the following HPE Broadcom-Based network adapters with bnxtnet driver version 212.0.119.0 (or prior) may experience a PSOD when high random I/O traffic is experienced:

HPE Ethernet 10/25Gb 2-port 631FLR-SFP28 Adapter
HPE Ethernet 10/25Gb 2-port 631SFP28 Adapter
HPE Ethernet 10Gb 2-port 535T Adapter
HPE Ethernet 10Gb 2-port 535FLR-T Adapter

•  The PSOD backtrace will have similar entries as below

            2018-12-06T14:41:28.012Z cpu7:2271232)@BlueScreen: #PF Exception 14 in world 2271232:vmkload_mod IP 0x418010952f01 addr 0x4313f20de5c4
PTEs:0x2040079023;0x0;
2018-12-06T14:41:28.012Z cpu7:2271232)Code start: 0x418010800000 VMK uptime: 0:15:14:58.979
2018-12-06T14:41:28.013Z cpu7:2271232)0x451aabe9b420:[0x418010952f01]DLM_free@vmkernel#nover+0x115 stack: 0x4309f20de5d0
2018-12-06T14:41:28.013Z cpu7:2271232)0x451aabe9b440:[0x418010950630]Heap_Free@vmkernel#nover+0x115 stack: 0x1
2018-12-06T14:41:28.014Z cpu7:2271232)0x451aabe9b490:[0x4180111e015e][email protected]#1+0x5b stack: 0x4cd74307c77a8257
2018-12-06T14:41:28.015Z cpu7:2271232)0x451aabe9b4b0:[0x4180111d4173][email protected]#1+0x108 stack: 0x1
2018-12-06T14:41:28.015Z cpu7:2271232)0x451aabe9b510:[0x41801082d296]Driver_QuiesceDevice@vmkernel#nover+0x63 stack: 0x41801082d291
2018-12-06T14:41:28.016Z cpu7:2271232)0x451aabe9b550:[0x4180108287ec]DeviceQuiesce@vmkernel#nover+0xed stack: 0x418010828854
2018-12-06T14:41:28.016Z cpu7:2271232)0x451aabe9b580:[0x41801082a2ad]DeviceTreeWalk@vmkernel#nover+0x106 stack: 0x41801082896a
2018-12-06T14:41:28.016Z cpu7:2271232)0x451aabe9b5e0:[0x41801082af1b]Device_ModuleUnload@vmkernel#nover+0x174 stack: 0x4307c77be9e0
2018-12-06T14:41:28.017Z cpu7:2271232)0x451aabe9b640:[0x4180109058fc]Mod_Unload@vmkernel#nover+0x59 stack: 0x1700000000
2018-12-06T14:41:28.017Z cpu7:2271232)0x451aabe9b6b0:[0x418010fa5144]UW64VMKSyscallUnpackModUnload@(user)#<None>+0x41 stack: 0x418010b2bf7a
2018-12-06T14:41:28.018Z cpu7:2271232)0x451aabe9bed0:[0x418010f3b660]User_UWVMK64SyscallHandler@(user)#<None>+0x249 stack: 0x0
2018-12-06T14:41:28.018Z cpu7:2271232)0x451aabe9bf30:[0x41801095f648]SyscallUWVMK64@vmkernel#nover+0x90 stack: 0x0
2018-12-06T14:41:28.018Z cpu7:2271232)0x451aabe9bf38:[0x418010960066]gate_entry@vmkernel#nover+0x67 stack: 0x0
2018-12-06T14:41:28.021Z cpu7:2271232)base fs=0x0 gs=0x418041c00000 Kgs=0x0

 

• The NIC driver and firmware versions that caused the issue

Firmware Version: 212.0.102.
bnxtnet driver Version: 212.0.119.0
 

Note:The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment.
Disclaimer:VMware is not responsible for the reliability of any data,opinions,advice or statements made on third-party websites.Inclusion of such links does not imply that VMware endorses,recommends or accepts any responsibility for the content of such sites

 

Environment

VMware vSAN 6.x

Cause

This PSOD is seen when high random I/O traffic is experienced on the reported NIC cards.

Resolution

To fix and prevent this issue, download and install for ESXi 6.0 and ESXi 6.5 bnxtnet version 214.0.190.0  and for ESXi 6.7 bnxtnet version 214.0.195.0

For steps to install the driver refer to VMware KB : How to download and install async drivers in ESXi 5.x/6.x 

For more information refer to HPE Customer Advisory   


Workaround:
No  workaround  available to avoid this issue.

Additional Information

Impact/Risks:
It requires reboot of ESXi hosts after installing the driver.