VMware NSX-T 3.2.2 Firewall rules not getting removed even if the VM is added into DFW exclusion list

Products

VMware vDefend Firewall

Issue/Introduction

You have upgraded to NSX-T 3.2.2.
DFW rules are being applied to VMs which have been added to the DFW exclusion list.
This is confirmed by the presence of a slot 2 when running the below command on the ESXi of the affected VM:
- Use the summarize-dvfilter command to list the DFW 2 VM filter:

[root@esx-03:~] summarize-dvfilter | grep Test_VM -A2

world 1087494 vmm0:TEST_VM vcUuid:'50 1d 7f 04 d3 39 6d 46-## ## ## ## ## ## ## ##'
port 67108877 NSX-T Manager 01.eth0
vNic slot 2 <------- A VM on the exclusion list will not have a slot 2 listed
name: nic-1087494-eth0-vmware-sfw.2

Using the slot 2 filter from above in the following command vsipioctl getrules -f <filter name> there are firewall rules prescent:

[root@esx-03:~] vsipioctl getrules -f nic-1087494-eth0-vmware-sfw.2
...
rule 23465 at 1 inout protocol any from addrset rsrc123456 to addrset rdst654321 goto_filter tag 'Test_Rule'

...

Checking from root of the NSX manager you see "entries(0)" for the Checkpoint Writer in /var/log/corfu/corfu-compactor-audit.log:

corfu-compactor-audit.5.log.gz:2022-12-16T17:00:20.379Z INFO main CheckpointWriter - appendCheckpoint: completed checkpoint for 6226ea59-####-####-####-cb9c42f9####, entries(0), cpSize(1) bytes at snapshot Token(epoch=1303, sequence=606268945) in 56 ms

NOTE: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware NSX 4.x
VMware NSX-T Data Center 3.x

Resolution

This issue is resolved in VMware NSX 3.2.4
This issue is resolved in VMware NSX 4.2.0

Workaround:
Perform an update on "/infra/settings/firewall/security" entity to encourage sync operation between MP to CCP. This can be done in 2 ways. For this case you can choose an attribute in settings entity to toggle e.g. say "Firewall Auto draft" field. It is recommend to not modify any security configuration when this operation is performed.

Via Manager UI:

Step 1: Navigate to Security -> Distributed Firewall -> Actions -> General Settings -> General Firewall -> Auto Save Drafts -> Toggle (set it OFF if its ON or set it ON if its OFF) -> Save
Step 2: Repeat Step1 to revert "Auto Save Drafts" setting i.e. to it previous state.

Via Policy APIs:

Step 1: Gather the body required to proform these steps:

GET https://<Manager IP>/policy/api/v1/infra/settings/firewall/security
{
    "idfw_enabled": false,
    "idfw_event_log_scraper_enabled": false,
    "idfw_loginsight_enabled": false,
    "resource_type": "DfwFirewallConfiguration",
    "id": "security",
    "display_name": "security",
    "path": "/infra/settings/firewall/security",
    "relative_path": "security",
    "parent_path": "/infra",
    "remote_path": "",
    "unique_id": "9cd8b89a-f7e3-473d-####-##########",
    "realization_id": "9cd8b89a-f7e3-473d-####-##########",
    "owner_id": "0ea7293f-a909-4a7c-####-##########",
    "origin_site_id": "0ea7293f-a909-4a7c-####-##########",
    "marked_for_delete": false,
    "overridden": false,
    "enable_firewall": true,
    "disable_auto_drafts": false, <------------- You will use this attribute to toggle (this value can be 'true' or 'false')
    "global_addrset_mode_enabled": true,
    "_create_time": 1662999213629,
    "_create_user": "system",
    "_last_modified_time": 1663098757027,
    "_last_modified_user": "admin",
    "_system_owned": false,
    "_protection": "NOT_PROTECTED",
    "_revision": 4
}

Step 2: Copy API response from Step 1 and use it as payload, toggle "disable_auto_drafts" value and invoke below API to update settings

PUT https://<Manager IP>/policy/api/v1/infra/settings/firewall/security

{
    "idfw_enabled": false,
    "idfw_event_log_scraper_enabled": false,
    "idfw_loginsight_enabled": false,
    "resource_type": "DfwFirewallConfiguration",
    "id": "security",
    "display_name": "security",
    "path": "/infra/settings/firewall/security",
    "relative_path": "security",
    "parent_path": "/infra",
    "remote_path": "",
    "unique_id": "9cd8b89a-f7e3-473d-####-##########",
    "realization_id": "9cd8b89a-f7e3-473d-####-##########",
    "owner_id": "0ea7293f-a909-4a7c-####-##########",
    "origin_site_id": "0ea7293f-a909-4a7c-####-##########",
    "marked_for_delete": false,
    "overridden": false,
    "enable_firewall": true,
    "disable_auto_drafts": true, <------------------------ Toggled this attribute to 'true'
    "global_addrset_mode_enabled": true,
    "_create_time": 1662999213629,
    "_create_user": "system",
    "_last_modified_time": 1663098757027,
    "_last_modified_user": "admin",
    "_system_owned": false,
    "_protection": "NOT_PROTECTED",
    "_revision": 4
}

Step 3: Copy API response from Step 2 and use it again as payload to revert back to original value.

PUT https://<Manager IP>/policy/api/v1/infra/settings/firewall/security
{
    "idfw_enabled": false,
    "idfw_event_log_scraper_enabled": false,
    "idfw_loginsight_enabled": false,
    "resource_type": "DfwFirewallConfiguration",
    "id": "security",
    "display_name": "security",
    "path": "/infra/settings/firewall/security",
    "relative_path": "security",
    "parent_path": "/infra",
    "remote_path": "",
    "unique_id": "9cd8b89a-f7e3-473d-####-##########",
    "realization_id": "9cd8b89a-f7e3-473d-####-##########",
    "owner_id": "0ea7293f-a909-4a7c-####-##########",
    "origin_site_id": "0ea7293f-a909-4a7c-####-##########",
    "marked_for_delete": false,
    "overridden": false,
    "enable_firewall": true,
    "disable_auto_drafts": false, <------------------------ Toggled this attribute back to 'false' (original value)
    "global_addrset_mode_enabled": true,
    "_create_time": 1662999213629,
    "_create_user": "system",
    "_last_modified_time": 1663098757027,
    "_last_modified_user": "admin",
    "_system_owned": false,
    "_protection": "NOT_PROTECTED",
    "_revision": 5
}

Once complete confirm the DFW exclusion list is now functioning as expected.