NSX Manager GUI intermittently showing services down resulting in web UI error code 101 or Cluster Degraded health checks due to /tmp usage at 100%.

Products

• VMware NSX

Issue/Introduction

• May intermittently see NSX GUI failing to load with "Error code: 101" and reporting some services are DOWN. 

• Running 'df -h' from the NSX Manager root command prompt shows the /tmp file system at 100% usage. 

Environment

VMware NSX

Cause

The /tmp file system usage can fill due to a variety of causes. One example was the generation of a large number of hidden files named .<user>_####_threaddump.tmp files cause by a stale TZP reference as detailed in KB 410402.

Resolution

If you believe you have encountered this issue, please open a support request with Broadcom Support and refer to this KB article.

Workaround:

• Perform a rolling reboot of the NSX Managers which will empty the /tmp directory.

Warning: This workaround will clear the /tmp usage, at least temporarily, but may also erase the evidence needed for Broadcom GS to determine what was causing /tmp to fill.