nsx_adf_collect.py fails with Error: No such file or directory: 'host_latency.json'

Products

• VMware NSX

Issue/Introduction

• Run nsx_adf_collect.py for example:
  python3 /opt/vmware/nsx-common/python/adf/nsx_adf_collect.py -i 10 -z adf_date_$(date "+%Y%M%d_%H%M%S").tar -a -o <output_path>
• nsx_adf_collect.py throws Error:
  Running ADF Collect, collecting 1 sample(s).
Traceback (most recent call last):
  File "/opt/vmware/nsx-common/python/adf/nsx_adf_collect.py", line 195, in <module>
    main()
  File "/opt/vmware/nsx-common/python/adf/nsx_adf_collect.py", line 173, in main
    sample = tools.run()
  File "/opt/vmware/nsx-common/python/adf/collector/host_tools.py", line 83, in run
    metadataLst.append(function())
  File "/opt/vmware/nsx-common/python/adf/collector/esx/esx_tools.py", line 60, in __runHostLatency
    metadata = self.runToolCmd('host-latency', cmd,
  File "/opt/vmware/nsx-common/python/adf/collector/host_tools.py", line 185, in runToolCmd
    os.remove(op)
FileNotFoundError: [Errno 2] No such file or directory: 'host_latency.json'
• /var/log/syslog.log has Error:
  <TIMESTAMP> Er(11) /opt/vmware/nsx-common/python/adf/collector/esx/host-latency.py: ERROR: Uncaught exception
<TIMESTAMP> Er(11)[+] /opt/vmware/nsx-common/python/adf/collector/esx/host-latency.py: Traceback (most recent call last):
<TIMESTAMP> Er(11)[+] /opt/vmware/nsx-common/python/adf/collector/esx/host-latency.py:   File "/opt/vmware/nsx-common/python/adf/collector/esx//host-latency.py", line 740, in <module>
<TIMESTAMP> Er(11)[+] /opt/vmware/nsx-common/python/adf/collector/esx/host-latency.py:     sw_info = switch_latency_sample(vswitch, args.count, args.duration,
<TIMESTAMP> Er(11)[+] /opt/vmware/nsx-common/python/adf/collector/esx/host-latency.py:   File "/opt/vmware/nsx-common/python/adf/collector/esx//host-latency.py", line 659, in switch_latency_sample
<TIMESTAMP> Er(11)[+] /opt/vmware/nsx-common/python/adf/collector/esx/host-latency.py:     run_vprobe_script(port_emt, duration, verbosity, retry)
<TIMESTAMP> Er(11)[+] /opt/vmware/nsx-common/python/adf/collector/esx/host-latency.py:   File "/opt/vmware/nsx-common/python/adf/collector/esx//host-latency.py", line 581, in run_vprobe_script
<TIMESTAMP> Er(11)[+] /opt/vmware/nsx-common/python/adf/collector/esx/host-latency.py:     returncode, stdout, stderr = runCmd(cmd_val)
<TIMESTAMP> Er(11)[+] /opt/vmware/nsx-common/python/adf/collector/esx/host-latency.py:   File "/opt/vmware/nsx-common/python/adf/collector/esx//host-latency.py", line 165, in runCmd
<TIMESTAMP> Er(11)[+] /opt/vmware/nsx-common/python/adf/collector/esx/host-latency.py:     prog = subprocess.Popen(
<TIMESTAMP> Er(11)[+] /opt/vmware/nsx-common/python/adf/collector/esx/host-latency.py:   File "/lib64/python3.11/subprocess.py", line 1026, in __init__
<TIMESTAMP> Er(11)[+] /opt/vmware/nsx-common/python/adf/collector/esx/host-latency.py:   File "/lib64/python3.11/subprocess.py", line 1851, in _execute_child
<TIMESTAMP> Er(11)[+] /opt/vmware/nsx-common/python/adf/collector/esx/host-latency.py: OSError: [Errno 24] Too many open files^@
• There are about 128 or more ports on the host
  To list ports on the host, run net-stats -l

Environment

VMware NSX

Cause

Some file descriptors are not closed while running nsx_adf_collect.py.

Resolution

Currently, there is no solution.

To workaround this issue, reduce a number of ports on the host for example:

• vMotion some running VMs to other hosts until about 120.
• Stop VMs
• Remove vNICs from VMs