High CPU Utilization in NSX Controllers

Products

• VMware NSX

Issue/Introduction

This article addresses network stability issues that may occur due to high CPU utilization within environments where NSX and vRNI (or other CLI polling software) are used together for data collection and analysis.

Symptoms:

• NSX Controllers show high CPU Utilization (80-100%).
• CPU Utilization Graphs on the vCenter for the Controller Appliance show a linear increase up to the current Value.
• vRealize Network Insight is used in the Environment, or any other software using CLI to periodically poll NSX Controllers.

Environment

NSX for vSphere 6.3.5

Cause

vRealize Network Insight uses a CLI method to pull relevant Network Information from the Controllers. This procedure creates a new log file per action on the NSX Controller, up to one new log file per second. The logrotate process is unable to cope with the large amount of individual log files created.

Note: vRNI is just an example for a Software triggering this issue; other VMware or third party Software, or simple scripts used to login to the NSX Controllers CLI in short intervals, may trigger the same behavior.

Resolution

This is resolved in NSX 6.3.6 and NSX 6.4.1

vRNI 3.7 uses a different mechanism to pull most information from the NSX Controllers, which reduces the amount of queries and therefore log files. A change in the logging mechanism for the CLI access will be included in an upcoming NSX Release.


Workaround:

• To prevent new log files from being generated, disable the polling of NSX Controllers.
• To relieve logrotate, VMware Support can help flush the existing log files on the NSX Controllers. (Please open a support request for this workaround.)
• CPU Utilization should go back to normal levels after a few minutes.
• Controller reboot or redeployment is not required.

Polling Information from the NSX Controllers can be disabled on vRNI with the following steps:

1. Go to Accounts and Data Sources page on Settings
2. Click on the Edit data source icon on right side of the NSX Manager.
3. Unselect the Enable NSX Controller.
4. Click on Submit button.

This will disable the collection of data from NSX Controller, however the data collection for other aspects of the environment will continue, including getting data from other components of  the NSX Manager.

Additional Information

Impact/Risks:

Controllers might fail, leading to potential network disruption.