Firewall publishing takes long time to complete causing high CPU on NSX Manager

Products

• VMware NSX Networking

Issue/Introduction

Identify the number of container updates sent towards host during publish.
Example:--
grep "2019-12-30" vsm.log* | grep "rule updates," | grep -v "0/0 container updates" | awk '{print $1,$2,$19,$20}'
2019-12-30 12:29:52.982 75/75 container
2019-12-30 12:30:17.294 301/301 container
2019-12-30 12:40:58.887 2/2 container
2019-12-30 12:41:12.935 98/98 container
2019-12-30 12:41:41.753 782/782 container
2019-12-30 12:09:30.828 4/4 container
2019-12-30 12:09:33.865 84/84 container
2019-12-30 12:10:03.585 695/695 container
2019-12-30 11:22:59.970 804/804 container
2019-12-30 11:11:22.003 6/6 container
2019-12-30 11:11:24.332 218/218 container
2019-12-30 11:12:27.263 152/152 container

Symptoms:
Publish failure is seen for some hosts in NSX plugin UI.
NSX Manager could show high CPU.

Cause

Each security tag is under multiple security groups exceeding the supported number.
As security tag is added/deleted for a VM, there are many container updates on hosts within a short span of time.
During this time, the UI might show "Publish failure" for some hosts or may take a long time to publish.

Resolution

This is a known issue affecting NSX for vSphere and is fixed in 6.4.11
"Fixed Issue 2667067: Translation of Security Groups with security tag as members takes a long time at scale, when 400+ security groups contain the same tag"

Workaround:

 

 

Additional Information

Impact/Risks:
Any new configuration changes will not be realized. There is no impact to the dataplane.