NSX Edge Upgrade Fails with 'download_os failed' Due to Insufficient Disk Space

Products

• VMware NSX

Issue/Introduction

• During an NSX upgrade, one or more Edge nodes fail to upgrade. The NSX Manager UI displays an error indicating that the Edge upgrade has failed and suggests checking error details to determine if manual resolution is needed before selecting 'Retry Upgrade'.

• An NSX upgrade fails during the Edge node upgrade step. The detailed error shows below

<version_file>/Edge/nub/VMware-NSX-edge-<version>.nub download OS task failed on edge TransportNode.

• When logged in to the affected NSX Edge Node CLI as root and performed the following checks , we could see that /image partion is at 100%

  df -h

Environment

VMware NSX

Cause

The /image partition on the Edge Transport Node has reached 100% utilization. This partition is used to store the NSX file-store and incoming upgrade image files (.nub). When the partition is full, the upgrade agent cannot extract the repository files required for the installation.

 

Resolution

/image partion verification :

To verify disk usage, log in to the affected NSX Edge Node CLI as root and perform the following checks:

df -h

Confirm if the /image mount point is at 100% capacity.

Identify Disk Consumers: Identify large files (such as support bundles, core dumps, or old upgrade files) consuming space:

du -h --max-depth=2 /image/ | sort -hr | head

Resolution

Clean Up Files: Remove any large, unnecessary files such as old support bundles or core dumps or the .pcap files from the /image directory.
Verify Space: Ensure df -h shows the partition is now below the high-utilization threshold to accomodate the image for Edge upgrade.
Re-trigger Upgrade: In the NSX Manager UI, click 'Retry Upgrade'.

Note: Once the NSX upgrade completes successfully, the Edge Transport Node is designed to purge these upgrade image files automatically.