NSX Upgrade precheck showing as failure in NSX UI, error ESXi host has less than 10% space on root partition.

Products

• VMware NSX

Issue/Introduction

During the NSX upgrade, the host precheck stage of the upgrade gives you a warning indicating a ESXi host or multiple ESXi hosts have failed due to less than 10% space on their root partition.  

 

Environment

VMware NSX

Cause

ESXi has less than 10% on the root partition, which is not sufficient to allow NSX components to be installed. 

Resolution

1. First determine the size of the root (/) partition by running the following command:

  df -h /

Output will be similar to:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3       2.5G  2.3G  200M  90%       /

If the overall Size looks smaller than expected, most likely the scratch partition is not setup on the host.

2. To determine this if the ESXi has a configured scratch partition or if the host is using a RAM drive. 

Run the following command to determine if scratch is correctly configured: 

esxcli system settings advanced list -o ScratchConfig.CurrentScratchLocation

Output will be similar to:

 

Path: /ScratchConfig/CurrentScratchLocation

Type: string

String Value: /tmp/scratch

• • • If this shows /tmp/scratch, ESXi is using a RAM disk.

    • If it shows a path on a datastore,/vmfs/volumes/...  it's using a persistent scratch partition.

If it does not have a scratch partition, then use the following KB 317689 to setup a persistent scratch partition on the ESXi host.  

 

If a scratch partition is configured and the space is low then first determine what is taking up space on ESXi root partition. 

 

3.  Run the following command to list the overall disk usage: 

 

df -h

 

Output will be similar to: 

Filesystem      Size  Used Avail Use%    Mounted on
VMFS-5          500G  200G  300G  40%   /vmfs/volumes/datastore1
/dev/sda3       2.5G  2.3G  200M  92%   /
/dev/sda1       250M   50M  200M  20%   /bootbank
/dev/sda2       250M   45M  205M  18%   /altbootbank

 

Look for high usage on / — anything above 90% is a red flag.

 

4. Next run the following command to determine which directories on root are taking the most space. 

du -h --max-depth=1 /

Output will be similar to: 

1.2M    /etc
45M     /var
8.0K    /home
200M    /tmp
2.3G    /

 

5. Run the following command to list size of the directories that were large in the last step. For example var. 

du -h --max-depth=1 /var

Output will be similar to: 

5.0M    /var/core
20M     /var/log
10M     /var/tmp
10M     /var/spool
45M     /var

6. Run the following to find the largest files 

find / -type f -size +100M -exec ls -lh {} \; | sort -k 5 -rh

Output will be similar to: 

-rw-------    1 root root  500M Oct  5 10:00 /var/core/vmkernel-zdump.001
-rw-r--r--    1 root root  150M Oct  5 09:00 /var/log/hostd.log
-rw-r--r--    1 root root  120M Oct  5 08:00 /tmp/large-temp-file.tmp

From the above we can see that the largest file showing is a log file which can be deleted to free up space. 

 

7. Run the following to determine the largest files in the log directory

ls -lh /var/log

Output will be similar to: 

-rw-r--r-- 1 root root  150M Oct  5 09:00 hostd.log
-rw-r--r-- 1 root root   80M Oct  5 08:00 vpxa.log
-rw-r--r-- 1 root root   10M Oct  5 07:00 vmkernel.log

 

8. We can truncate the hostd.log in this example to free up space. Run the following command to do so. 

cat /dev/null > /var/log/hostd.log

Clear up any other files that are large

 

If you are not sure on what files can be cleared or have determined that other files are taking up the space.

Please open up a support case with support to investigate further, mentioning this KB 413078.