vSphere HA agent failure / invalid on ESXi host due to vmware-fdm agent running out of memory

Products

• VMware vSphere ESX 7.x
• VMware vSphere ESX 8.x

Issue/Introduction

• ESXi hosts are running Cisco NFNIC drivers for storage.
• HA status is invalid and attempting to enable HA on the cluster will not succeed.
• In an HA and DRS enabled environment, a mass migration event can also occur which can cause resource contention. This is due to a failure of the vmware-fdm vib on some of the hosts inside the cluster and DRS then attempting to migrate VMs to hosts in which the vSphere HA agent is responding. 
• When investigating the /var/log/vobd.log file on the hosts, logs like the following where the T# (Target) continually increases each time the path is removed and re-added.:
  

  YYYY-MM-DD HH:MM:SS:MS: [scsiCorrelator] 5639277608251us: [vob.scsi.scsipath.add] Add path: vmhba2:C0:T583:L254
  YYYY-MM-DD HH:MM:SS:MS: [scsiCorrelator] 5639315873118us: [vob.scsi.scsipath.add] Add path: vmhba2:C0:T584:L254
  YYYY-MM-DD HH:MM:SS:MS: [scsiCorrelator] 5639492899791us: [vob.scsi.scsipath.remove] Remove path: vmhba2:C0:T578:L254
  YYYY-MM-DD HH:MM:SS:MS: [scsiCorrelator] 5639573618367us: [vob.scsi.scsipath.remove] Remove path: vmhba2:C0:T540:L254

• This constant iteration of the vmhba paths causes the hostAgentStats-20.stats file in the /var/lib/vmware/hostd/stats folder to grow unchecked.  Once the hostAgentStats-20.stats grows to too large of a size the vmware-fdm vib will crash. The /var/log/fdm.log will show similar log messages as:
  

  YYYY-MM-DD HH:MM:SS:MS error fdm[5174372] [Originator@6876 sub=Default opID=SWI-41a7] Unable to allocate memory
  YYYY-MM-DD HH:MM:SS:MS panic fdm[5174372] [Originator@6876 sub=Default opID=SWI-41a7]
  -->
  --> Panic: Unable to allocate memory
  --> Backtrace:
  --> [backtrace begin] product: VMware Fault Domain Manager, version: 7.0.3, build: build-24024786, tag: fdm, cpu: x86_64, os: linux, buildType: release
  --> backtrace[00] fdm[0x00E560C9]
  --> backtrace[01] fdm[0x00DD2305]
  --> backtrace[02] fdm[0x00D10649]
  --> backtrace[03] fdm[0x00D7AFEA]

Environment

• ESXi 7.0.x
• ESXi 8.0.x

Cause

This issue is attributed to a known problem where, in the presence of storage path flapping events, the Cisco NFNIC drivers, in conjunction with the vmware-fdm VIB, exhibit a behavior of incrementing the storage path number instead of re-establishing connectivity on the existing, original path identifier.

Resolution

The following steps will provide a temporary workaround to resolve the vsphere-fdm agent failure.

1. Disable HA in the cluster where the affected hosts reside.
   In vSphere Client, Select Cluster Object > Configure > vSphere Availability > Edit > Turn off vSphere HA.
   
   
2. Set DRS automation to Manual.  (This prevents potential issue of mass migration of VMs due to vSphere HA agent crash on host)
   In vSphere Client, Select Cluster Object > Configure > vSphere DRS > Edit > Automation Level > Set to Manual
   
   
3. Connect to each of the affected hosts via SSH.
   
   
4. Stop hostd on the host
   
   /etc/init.d/hostd stop

5. On the ESXi host, navigate to and move the files in: /var/lib/vmware/hostd/stats

   cd /var/lib/vmware/hostd/stats

   mv * /tmp/ 

Note: This will move respective files (hostAgentStats-20.stats hostAgentStats-metadata.xml hostAgentStats.idMap hostAgentStats.xml) to /tmp

6. Start hostd
   
   /etc/init.d/hostd start

   Note: Upon start, hostd will detect files are missing in /var/lib/vmware/hostd/stats and will recreate them. 

7. Once all affected hosts have completed above steps, re-enable HA and DRS in the cluster.
   
   
8. Once everything is restored after restarting hostd service, clean up the files from /tmp.