Unable to publish NSX-V DFW rules after upgrade to v6.4.11. The progress indicator spins forever with the message "Publishing rules" in the NSX dashboards

Products

• VMware NSX Data Center for vSphere

Issue/Introduction

Symptoms:

• Unable to publish the DFW rule after upgraded to v6.4.11, 6.4.12 and 6.4.13. The progress indicator spins in a circle forever.
• The IP Address information is missing for the Virtual Machines in the Security Groups objects. There is an error message saying "Internal server error has occurred"

• In the vsm.log file, you see entries similar to:

2021-10-21 04:25:01.062 GMT ERROR TaskFrameworkExecutor-23 FirewallMessagingManager:179 - - [nsxv@6876 comp="nsx-manager" errorCode="MP100" level="ERROR" subcomp="manager"] Exception while publishing rule set to cluster: domain-c1006.
2021-10-21 04:25:01.064 GMT INFO TaskFrameworkExecutor-23 EventHelper:172 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] SysEvent-Detailed-Message :(Kept only in logs) :: java.lang.NullPointerException
2021-10-21 04:25:01.066 GMT INFO TaskFrameworkExecutor-23 EventServiceImpl:119 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] [SystemEvent] Time:'Thu Oct 21 04:25:01.063 GMT 2021', Severity:'Critical', Event Source:'domain-c1006', Code:'301503', Event Message:'Failed to publish firewall configuration version 1634114508587 to cluster domain-c1xxx. Refer logs for details.', Module:'vShield Firewall', Universal Object:'false'

 

Firewall Publish failures:

2022-12-01 00:01:03.455 UTC ERROR TaskFrameworkExecutor-19 FirewallMessagingManager:236 - - [nsxv@6876 comp="nsx-manager" errorCode="MP100" level="ERROR" subcomp="manager"] Exception while publishing container set to cluster: domain-c1789xxx.
2022-12-01 00:01:21.237 UTC ERROR TaskFrameworkExecutor-2 FirewallMessagingManager:236 - - [nsxv@6876 comp="nsx-manager" errorCode="MP100" level="ERROR" subcomp="manager"] Exception while publishing container set to cluster: domain-c1789xxx.

Around the same time , we can see some containers are being updated.

2022-12-01 00:01:02.414 UTC INFO DCNPool-3 NotificationProcessor$ContextElement:166 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Adding container update for object securitygroup-36 with object generationNumber 1669852862xxx for context globalroot-0
2022-12-01 00:01:02.418 UTC INFO DCNPool-4 NotificationProcessor$ContextElement:166 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Adding container update for object securitygroup-1370 with object generationNumber 166985286xxxx for context globalroot-0
 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX Data Center for vSphere 6.4.x

Cause

 

• This issue will be observed only in case of IP address translations for SecurityGroups containing securityTags as members.
• The Security Group definitions has a dynamic criteria, and this criterion has tags/VM’s added to it.
• Each of the VM’s has vNIC’s, and against each vNIC we will learn IP address associated with that vNIC, either via VMware tools or IP discovery.
• If any of the vNIC doesn’t have an IP address associated with it, it will lead to a publish error, as above, this is due to the vsm config property set to true for property “translation.optimized.query.enable”. This property is introduced in new releases. 
• By moving the value of this property to false , the Security Group will ignore any vNIC which doesn’t have an IP address associated, which will result in DFW publish to go through.

Resolution

• This is a known issue affecting VMware NSX for vSphere 6.4.11 - 6.4.13. Fix is planned for 6.4.14

Workaround:

1. Log in to the NSX Manager through SSH as root.
2. Type in "st e".
3. change translation.optimized.query.enable=true to translation.optimized.query.enable=false in the file /home/secureall/secureall/sem/WEB-INF/spring/vsmConfig.properties
4. Restart NSX-V manager

Additional Information

Impact/Risks:

• New DFW rules will fail to publish.
• Other environmental impacts such as Edge Upgrades will fail until the publish failures are remediated.