Firewall rules may not get realized on edge when the interval between firewall rule publish operation is short

Products

• VMware vDefend Firewall

Issue/Introduction

Adding around 2500 firewall rules sequentially one at a time in a gateway policy with shorter time gap between each firewall rule could cause last few firewall rules in sequence may not get realized on edge.

Environment

NSX 9.0

Cause

NSX Manager misses to create tasks to process the user intents like adding gateway firewall rule due to the guards in place to avoid unbounded tasks getting created in system. 

Intent APIs like adding gateway firewall rule respond with 200 OK as soon as possible. NSX Manager task works asynchronously using markers per such each intents and takes lock on the gateway policy to make sure the integrity of the gateway policy with all rules stays intact when published to edge. Tasks can create new marker in case lock can not be taken and puts guards in place to avoid unbounded markers getting created in system. Due to this guard in system its possible markers might not get created when the number of markers are already at high limit.

Below is example of log showing there was missing lock of 1 policy. This does not mean there will be a missing rule for sure.

/var/log/proton/nsxapi.log

nsxapi.log:2025-01-31T00:06:40.252Z  INFO providerTaskExecutor-1-70 AbstractRuleSectionProvider 87123 POLICY [nsx@4413 comp="nsx-manager" level="INFO" subcomp="manager"] GatewayRuleSectionProvider handlePolicyChangeInternal - Completed with sections Processed 0, Missing lock 1, New DOM 1, Time taken for DOM 2244 ms

Resolution

Workaround:
User can update the gateway policy by changing info like display name/description to trigger syncing rules to edges.