High CPU Utilization in NSX Controllers

Products

• VMware NSX Networking

Issue/Introduction

• NSX Controllers show high CPU Utilization.
• CPU Utilization Graphs on the vCenter for the Controller Appliance show a linear increase up to the current Value.
• vRealize Network Insight might be used in the environment.
• The top command (with option 'R' to sort for Utilization), executed on the NSX Controller root shell, shows that the CPU is consumed almost entirely by various logrotate processes.
• The command strace -p <logrotate-pid> -f -o /var/tmp/strace_out on the NSX Controller root shell shows that logrotate is working on files located in /var/log/nvpcli.

Cause:

vRealize Network Insight uses a CLI method to pull relevant Network Information from the NSX Controllers. For example, it logs in to the NSX Controller, runs and parses a cli command, logs off again.
This procedure creates a new log file per action on the NSX Controller, up to one new log file per second. The logrotate process is unable to cope with the large amount of individual log files created.

Note: vRNI is just an example for a software triggering this issue; other VMware or third party software, or simple scripts used to log in to the NSX Controllers in short intervals, would trigger the same behavior.

Environment

VMware NSX for vSphere 6.3.x

Resolution

This issue is resolved in:
VMware NSX for vSphere 6.4.1, available at VMware Downloads.
VMware NSX for vSphere 6.3.7, available at VMware Downloads.



To work around this issue if you do not want to upgrade:

1. Disable the vRNI Controller polling (or other Software / Scripts using the CLI method to fetch Information from the Controllers); this should prevent the creation of the large volume of log files.
2. Delete the log files from /var/log/nvpcli periodically. CPU Utilization should go back to normal levels after a few minutes. Controller reboot or redeployment is not required.

Polling Information from the NSX Controllers can be disabled on vRNI with the following steps:

1. Navigate to Accounts and Data Sources page on Settings.
2. Click on the Edit data source icon on right side of the NSX Manager.
3. Unselect the Enable NSX Controller.
4. Click on Submit button.