Intermittent network connectivity loss and RDP disconnects of VM's caused by iSCSI storage heartbeat timeouts on ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual Machines experience intermittent network connectivity issues, including RDP disconnections and ping drops, while residing on an ESXi host.

• Packet captures show incoming ARP requests being sent to the VM, but no ARP replies returning by the VM during the incident.

• The VM Web Console appears hung or unresponsive during the connectivity loss.

• vmkwarning.log shows Connection reset by peer from the storage target

• vobd.log shows esx.problem.vmfs.heartbeat.timedout for the affected volume

• hostd.log captures PopulateCache failed and Lost access to volume errors during these windows, preventing the VM from completing I/O operations

Environment

VMware vSphere ESXi

Cause

• The issue is caused by an intermittent loss of communication between the ESXi iSCSI initiator and the storage target.
• During this stun period, the Guest OS is unable to process network traffic, leading to the observed ARP reply delays and RDP disconnects.

Resolution

1. Validate if the iSCSI vmk's are able to reach the storage target fine with no intermittent drops and validate if there are configuration mismatches in regards to MTU and jumbo frames (if using) across the entire network path (VMkernel, physical switches, and storage controllers) and fix the networking issues (if any identified).
   
   
2. Once the networking path is confirmed to be working fine, please review storage target logs with the storage vendor to identify why the array is terminating sessions or failing to respond to requests.

If the issue persists or if you need further assistance from Broadcom Support Team, feel free to open a case for further investigation.

Additional Information

• Contact Broadcom Support
• How to use pktcap-uw to troubleshoot network issues
• Connection to iSCSI target fails due to network failure