VMs residing on a host rebooted abruptly.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

Two VMs were rebooted on a host abruptly.

You may see these messages 'NFS: 6284: Status: No connection. Retrying synchronous write I/O 4 of 25 times' on /var/run/log/vmkernel.log

Environment

VMware ESXi 7.x

Cause

• The issue was primarily caused by the simultaneous failure of vmnic1 and vmnic4, which resulted in a network disruption.

• This disruption triggered NFS storage I/O errors, ultimately leading to a loss of connectivity for the VMs and forcing them to reboot.

/var/run/log/vmkernel.log shows the below:

2025-02-19T06:42:45.017Z cpu0:2097726)NFS: 6284: Status: No connection. Retrying synchronous write I/O 4 of 25 times

/var/run/log/vmkwarning.log:

2025-02-19T06:42:26.016Z cpu50:2100636 opID=79d630c1)WARNING: NFS: 2581: Failed to get attributes (I/O error)
2025-02-19T06:42:33.016Z cpu40:3915681 opID=218e57b4)WARNING: NFS: 2581: Failed to get attributes (I/O error)

• Failover from vmnic0 to vmnic5 is not recorded in logs.

2025-02-19T14:12:13.214Z: [netCorrelator] 16175198449711us: [vob.net.dvport.uplink.transition.down] Uplink: vmnic4 is down. Affected dvPort: 5672/50 09 89 6a 88 23 23 42-08 d6 82 fb 61 3b 97 75. 1 uplinks up. Failed criteria: 128
2025-02-19T14:12:13.214Z: [netCorrelator] 16175198449805us: [vob.net.vmnic.linkstate.down] vmnic vmnic4 linkstate down
2025-02-19T14:12:13.220Z: [netCorrelator] 16175198456706us: [vob.net.dvport.uplink.transition.down] Uplink: vmnic1 is down. Affected dvPort: 56/50 09 7e d3 68 46 0a 9f-6d f5 bf 34 0a 90 e6 09. 1 uplinks up. Failed criteria: 128

• Firmware version mismatch for the drivers being used as per the Broadcom compatibility Guide: Driver and firmware compatibility

vmnic4  0000:af:00.0     Up    25000  Full    9000  nmlx5_core  4.22.73.1004    14.28.4512        b8:xx:xx:xx:xx:d8  15b3  1015  15b3  0016  Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
vmnic5  0000:af:00.1     Up    25000  Full    1500  nmlx5_core  4.22.73.1004    14.28.4512        b8:xx:xx:xx:xx:d9  15b3  1015  15b3  0016  Mellanox Technologies MT27710 Family [ConnectX-4 Lx]


Resolution

• Ensure the network adapters are running the recommended firmware version as specified in the compatibility guide Driver and firmware compatibility

• Investigate the physical network infrastructure for any issues that could have caused the simultaneous failure of vmnic1 and vmnic4.

• Involve the NFS storage vendor to check for any issues related to the I/O errors and ensure that the NFS configuration is optimal.