NSX upgrade failures due to insufficient free space in the /tmp filesystem

Products

• VMware NSX
• VMware SDDC Manager / VCF Installer

Issue/Introduction

• NSX ESXI (TN) Upgrade
• Upgrade Error occurred - Check for sufficient free space on /tmp partition
• Esxi syslog:  
  

  "Error occurred while executing download_lcp script: [Errno 28] No space left on device\n"

Environment

• VMware SDDC Manager
• NSX 4.2.X
  
  

Cause

• The host_cpu_mem_daemon.log is an NSX tool to help with system capacity planning, installed on each ESXi host.
• This failure can occur on any ESX host that has been running the NSX VIB for an extended period (typically several weeks or longer) without a host reboot or NSX agent restart. 
• The /tmp ramdisk on impacted ESXi hosts is consumed by excessive logging from the host_cpu_mem_daemon.log. This log file accumulates data rapidly due to a known logging defect, leaving less than the required 180MB of free space needed for the NSX-T VIB deployment and staging process.

Resolution

This is a known issue impacting VMware NSX.

Workaround:

Before initiating or retrying a host upgrade, SSH into each affected ESX host as root user and verify available space. If /tmp has less than ~180 MB free, truncate the daemon log to reclaim the space.

1. Check available space

vdf -h | grep tmp

Example output on an affected host:

tmp 256M 77M 178M 30% --

2. Check the log file size

ls -lh /tmp/host_cpu_mem_daemon.log

3. Truncate the log file

   (The daemon keeps running; it re-creates the file)

> /tmp/host_cpu_mem_daemon.log

4. Confirm free space is now ≥ 190 MB

vdf -h | grep tmp

5. Retry the host upgrade from the NSX Manager UI.

Note: Do not delete the log file outright while the daemon is running; truncating with > is safe because the daemon holds the file open. If you delete it, the daemon will continue writing to the unlinked inode (consuming ramdisk invisibly) until it is restarted.