VM datastores corrupting

Products

• VMware vCenter Server

Issue/Introduction

• The host lost its connection to the storage network.
• The storage connectively loss was causing vmdk to become corrupt. 

Cause

• The physical NICs on the host are malfunctioning.
• The NICs are continuously crashing and generating firmware dumps in vmkernal.log.
• The NICs are running on outdated firmware and an outdated driver.
• A check against the VMware Hardware Compatibility List (HCL) confirmed that there is a mismatch and that newer, certified versions of both the firmware and driver are available.
• This incompatibility is causing the firmware to crash, leading to network and storage connectivity loss.

 

Resolution

1. Begin with the affected host.
   
   
2. Download the corresponding HCL-certified driver from the VMware support site. 
   
   Download and install async drivers in VMware ESXi
   
   
3. Install the firmware upgrade first.
   
   
4. Install the new driver to ensure it matches the newly installed firmware, as running older firmware with a newer driver can cause issues.
   
   
5. Reboot the host and monitor to confirm that the firmware dumps have stopped and network stability is restored.
   
   
6. Once confirmed, schedule and perform the same upgrades on the other hosts in the cluster to prevent the issue from recurring, as all hosts are likely running the same older drivers.