NSX fails to create Logical Routers due to stale firewall section priority assignments in Corfu

Products

• VMware NSX
• VMware vDefend Firewall

Issue/Introduction

• NSX fails to deploy Logical Routers
• CreateLogicalRouter operations are seen failing in /var/log/syslog on NSX Manager:
  
  20##-##-##T##:##:##:###Z <mgr hostname> NSX 18288 ROUTING [nsx@6876 audit="true" comp="nsx-manager" level="INFO" reqId="<UUID>" subcomp="manager" update="true" username="tas"] UserName="tas", ModuleName="LogicalRouter", Operation="CreateLogicalRouter", Operation status="failure", New value=[{"router_type":"TIER1","policy_owned":false,"display_name":"<name>","tags":[{"scope":"ncp/version","tag":"1.2.0"},{"scope":"ncp/cluster","tag":"<name>"},{"scope":"external_id","tag":"<UUID>"},{"scope":"ncp/cf_org_guid","tag":"<UUID>"},{"scope":"ncp/cf_org_name","tag":"<name>"}]}]
  
  
• Searching for 'router' and 'POST' in /var/log/proxy/localhost_access_log* shows 400 Bad Request responses:
  
  20##-##-##T##:##:##:###Z <IP> - "POST /api/v1/logical-routers HTTP/1.1" 400 190 329 328
20##-##-##T##:##:##:###Z <IP> - "POST /api/v1/logical-routers HTTP/1.1" 400 190 275 274
20##-##-##T##:##:##:###Z <IP> - "POST /api/v1/logical-routers HTTP/1.1" 400 190 56 56
20##-##-##T##:##:##:###Z <IP> - "POST /api/v1/logical-routers HTTP/1.1" 400 190 47 46
20##-##-##T##:##:##:###Z <IP> - "POST /api/v1/logical-routers HTTP/1.1" 400 190 31 30

• There are "Bulk priority operation executed with status success=false" messages in /var/log/proton/nsxapi.log:
  
  20##-##-##T##:##:##:###Z  INFO FIREWALL_UFO_PRIORITY_PROCESSOR-0 AbstractPersistedQueueProcessor 4486 SERVICE [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Bulk priority operation executed with status success=false for configId : FirewallConfiguration/e6b33ca8-####-####-####-0242ac130013, having 17 operation(s) in 3693219 ms
20##-##-##T##:##:##:###Z  INFO FIREWALL_UFO_PRIORITY_PROCESSOR-0 AbstractPersistedQueueProcessor 4486 SERVICE [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Bulk priority operation executed with status success=false for configId : FirewallConfiguration/e6b33ca8-####-####-####-0242ac130013, having 17 operation(s) in 3253448 ms
20##-##-##T##:##:##:###Z  INFO FIREWALL_UFO_PRIORITY_PROCESSOR-0 AbstractPersistedQueueProcessor 4486 SERVICE [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Bulk priority operation executed with status success=false for configId : FirewallConfiguration/e6b33ca8-####-####-####-0242ac130013, having 17 operation(s) in 3218642 ms
  
  
•  The Corfu table nsx$FirewallSectionPriority with UUID 4b382bcb-####-####-####-79d5ad5035a9 has many entries (175546 in below example)    
  
  grep 'completed' var/log/corfu/corfu-compactor-audit.log | sort -n -k 2 -t'(' | tail 

2023-07-24T01:21:37.543Z  INFO main CheckpointWriter - appendCheckpoint: completed checkpoint for 4b382bcb-####-####-####-79d5ad5035a9, entries(175546), cpSize(65605992) bytes at snapshot Token(epoch=2638, sequence=5489535492) in 78233 ms

Cause

Firewall Sections are getting deleted as Logical Routers are created and deleted over time, but section priority entries in Corfu are not cleaned up.

Resolution

This issue is resolved in VMware NSX 4.1.1
This issue is resolved in VMware NSX 4.2.0

Workaround:
Contact Broadcom Support

Additional Information

Impact/Risks:
NSX fails to deploy Logical Routers.