/tmp on ESXi hosts configured for NSX-T may become heavily utilized or full

Products

• VMware NSX

Issue/Introduction

Symptoms:
On an ESXi host configured for NSX-T Data Center 2.4.x

• Scratch is set to default /tmp/scratch
• The output of the vdf command shows /tmp is heavily utilized or full
• There are multiple log files like this consuming space

          /tmp/scratch/log/nsx/metrics/proto_2019_09_01T16-29-56.log

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 2.x

Cause

NSX-T creates proto_<timestamp>.log files in scratch/log/nsx/metrics/. These are binary files which contain metrics collected from the different NSX-T daemons on the ESXi host. These files are intended for use by Engineering when troubleshooting issues on the host.

Resolution

This is a known issue impacting NSX-T 2.4.x and will be resolved in a future version of NSX-T Data Center.

Workaround:
It is recommended to configure a scratch partition on ESXi.
If this is not an option e.g. if VSAN is the only storage available then the following workarounds can be used.

Option 1:
Present another storage device to the ESXi hosts e.g. NFS/iSCSI datastore and configure scratch there

or

Option 2:
Enforce a stricter log rotation policy to ensure these files consume less storage space when located in /tmp

1) chmod +t /etc/vmware/nsx-opsagent/nsx-opsAgent.xml
2) Edit /etc/rc.local.d/local.sh.
   Add the following 3 lines below the comments section and above the line "exit 0"

   chmod +t  /etc/vmware/nsx-opsagent/nsx-opsAgent.xml
  sed -i 's/<max_size>1 GiB<\/max_size>/<max_size>40 Mib<\/max_size>/; s/<max_files>50<\/max_files>/<max_files>2<\/max_files>/' /etc/vmware/nsx-opsagent/nsx-opsAgent.xml
  /etc/init.d/nsx-opsagent restart

3) Execute the config change by running the updated script
   #/etc/rc.local.d/local.sh

Note, this change will persist ESXi host reboot.