vSphere HA State – HA Agent Unreachable: Cannot install the vCenter Server agent service. "Cannot upload agent"

Products

• VMware vSphere ESX 6.x
• VMware vSphere ESX 7.x
• VMware vSphere ESX 8.x

Issue/Introduction

This article provides information on how to troubleshoot and resolve situations where the /tmp partition becomes full on ESXi, preventing the vSphere HA (Fault Domain Manager) agent (vmware-fdm) from being installed or updated.

Symptoms

• Unable to install or update the vCenter Server vSphere High Availability (vSphere HA) agent service.
• The vmware-fdm VIB (the HA agent package) fails to install on ESXi.
• In the vCenter Server summary, the following error may be displayed:
  vSphere HA agent for this host has an error: vSphere HA agent cannot be correctly installed or configured.
  
  
• Within the ESXi Host logs under the var/log/vmkernel.log, you may see messages such as:
  YYYY-MM-DDTHH:MM:SS.6972 cpu34:2101938 WARNING: VisorFSRam: 203: Cannot extend visorfs file /tmp/VMware_bootbank vmware-fdm 8.0.3-1.24853646.vib because its ramdisk (tmp) is full.
YYYY-MM-DDTHH:MM:SS.6972 cpu34:2101938 WARNING: VisorFSRam: 203: Cannot extend visorfs file /tmp/VMware_bootbank vmware-fdm 8.0.3-1.24853646.vib because its ramdisk (tmp) is full.
  
  
• The /tmp folder shows 90% or higher usage:
  vdf -h
tmp   192M   192M   0B   100%

Environment

• VMware vSphere ESXi 6.7
• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0

Cause

The issue occurs because of inadequate file cleanup on the ESXi host.

One or more RAM disks that make up the ESXi root filesystem can reach resource limits due to:

• Software/service failures.
• Incorrect logging paths.
• Large or excessive temporary files filling up /tmp.

Resolution

1. Check whether the /tmp directory is full
   vdf -h| grep tmp
   
   
2. Identify large files
   cd /tmp
du -a . | sort -n -r | head -n 20
   
   
3. Examine suspicious files/directories
   ls -lha
   
   
4. Free up space by removing unnecessary or old log files. Refer to the below table for files commonly filling /tmp

Large/frequent files name	Resolution
mili2d.log	Delete the file Restart management services  - Full ramdisk '/var' causes /var/log/EMU/mili/mili2d.log to stop logging
ahs.debug	The event log of HPE ProLiant hosts shows error: The ramdisk 'tmp' is full. As a result, the file /tmp/auto-backup.1619198/local.tgz could not be written
ql_ima.log ql-ima_sdm.log ql_ima_sdm.log.old	Update the QLogic driver to latest version, available at Broadcom Support Workaround Remove the concerned ql_ima.log files and restart the management agents. For information on installing async drivers, see Download and install async drivers in VMware ESXi (broadcom.com)

5. If needed, reboot host
   • Place the host into maintenance mode.
   • Reboot to release file handles and fully clear /tmp.
6. Validate HA agent installation
   • Once /tmp has free space, vCenter automatically uploads and installs the vmware-fdm VIB.
     To confirm version:
     esxcli software vib list | grep vmware-fdm

Additional Information

Impact/Risks

• If /tmp is completely full, the ESXi host may fail to update critical files and could become unresponsive in vCenter.
• HA configuration and agent deployments will fail until space is freed.