ESXi host addon installation fails with "Errno 28 No space left on device" and VIB checksum errors due to /altbootbank corruption.

Products

• VMware vSphere ESXi

Issue/Introduction

• Below stated Task observed in host:
  Task Name - Install
  Status - An error occurred during host configuration.
  Initiator - com.vmware.vr-sa-########-####-####-#####-#######@vsphere.local
• Within the /var/log/esxupdate.log file on the affected ESXi host, we observe checksum verification failures followed by an inability to find the VIB:
  

  yyyy-mm-ddThh:mm:ssZ esxupdate: 5528180: vmware.esximage.Vib: DEBUG: Verifying VIB VMware_bootbank_vmware-hbr-agent_8.0.3-0.0.XXXXXXX signature #2
  yyyy-mm-ddThh:mm:ssZ esxupdate: 5528180: BootBankInstaller.py: ERROR: Failed to verify checksum for payload vmware-h: VMware_bootbank_vmware-hbr-agent_8.0.3-0.0.24299508: Cannot find VIB VMware_bootbank_vmware-hbr-agent_8.0.3-0.0.XXXXXX in BootbankInstaller

• Immediately following the checksum error, the logs indicate a staging failure with a disk space error, despite df -h showing adequate free space on the partition:
  

  yyyy-mm-ddThh:mm:ssZ esxupdate: 5538802: esxupdate: ERROR: vmware.esximage.Errors.InstallationError: Failed to copy files from /tmp/stagebootbank to /altbootbank: [Errno 28] No space left on device: '/altbootbank/xorg.v00'

• Running ls -la /altbootbank via an SSH session returns junk characters or garbled text instead of standard file names.

Environment

VMware vSphere ESXi 7.x, 8.x, 9.x

Cause

This issue is caused by vFAT file system corruption on the physical partition backing the /altbootbank directory.

ESXi relies on the alternate bootbank to stage the new OS image or addon payload. When the file allocation table (FAT) index on the boot device becomes corrupted, the OS cannot read the directory structure or write new files. The ESXi kernel misinterprets this inability to write as the disk being completely out of space (triggering [Errno 28] No space left on device). This prevents the BootBankInstaller from extracting and copying the VIBs, causing the checksum and installation tasks to fail.

Resolution

To resolve this issue, you must repair or rebuild the corrupted /altbootbank partition. Connect to the affected ESXi host via SSH with root privileges and execute the following steps:

Step 1: Identify the Partition

Identify the exact physical device path backing the /altbootbank directory:

vmkfstools -P /altbootbank

Note the deviceFullPath in the output (e.g., /dev/disks/mpx.vmhba32:C0:T0:L0:6). Ensure you capture the correct partition number (typically 6).

Step 2: Attempt File System Repair

Run the built-in DOS file system checker in auto-repair mode against the identified partition:

dosfsck -a -w /dev/disks/<your-disk-and-partition-id>

Once the repair completes, run ls -la /altbootbank. If the output displays clean, readable file names, the repair was successful. You can retry the addon installation from vCenter.

Step 3: Format and Rebuild (If Repair Fails)

If dosfsck fails or the junk characters remain, you must manually format the partition and clone the healthy OS image from the active /bootbank.

1. Format the corrupted partition: (Warning: Ensure you are strictly targeting the altbootbank partition ID identified in Step 1 to avoid data loss).

   vmkfstools -C vfat /dev/disks/<your-disk-and-partition-id>
   

2. Copy the active bootbank contents to the altbootbank:

   cp -r /bootbank/* /altbootbank/
   

3. Verify the rebuild:

   ls -la /altbootbank
   

   The output should now match the clean files in your active bootbank. You may now retry the installation via vSphere Lifecycle Manager.

Step 4: Hardware Replacement

If the formatting step fails or the vFAT corruption immediately returns after a reboot, the underlying physical boot media (SD card, USB drive, or local disk) has likely suffered a hardware degradation or failure. The host must be evacuated and reinstalled on healthy boot media.