DFW Policies in Failed Status with Error Code 1101 - "Failed to process commands by firewall kernel"
search cancel

DFW Policies in Failed Status with Error Code 1101 - "Failed to process commands by firewall kernel"

book

Article ID: 410585

calendar_today

Updated On:

Products

VMware vDefend Firewall VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

  • DFW Policies are in a "Failed" status in the NSX UI
  • When clicking the "Failed" realization status, you can see the following error for several hosts:

    [Error Code = 1101, Error Message = 'Failed to process commands by firewall kernel'. Affected Entities = '{}']





  • Issue the following command as root user on an impacted ESXi host:

    /opt/vmware/nsx-nestdb/bin/nestdb-cli --beautify --json --cmd get vmware.nsx.nestdb.GlobalConfigMsg | grep global_addrset_mode_enabled

    The returned value for global_addrset_mode_enabled is false


  • The following is seen in vmkernel.log on the ESXi host:

    2025-09-11T13:18:44.913Z In(182) vmkernel: cpu82:2100975)pfr_attach_table: nic-#######-eth0-vmware-sfw.2: ERROR ***************** local root table <Address Set UUID> not found

    2025-09-11T13:18:44.913Z In(182) vmkernel: cpu82:2100975)pfioctl: DIOCADDRULE failed with error 22

    2025-09-11T13:18:44.913Z In(182) vmkernel: cpu82:2100975)VSIPConversionCreateRuleSet: Cannot insert ## rule #######: 22

    2025-09-11T13:18:44.913Z In(182) vmkernel: cpu82:2100975)pf_rollback_rules: rs_num: 1, anchor: mainrs

    2025-09-11T13:18:44.913Z In(182) vmkernel: cpu82:2100975)pf_rollback_rules: rs_num: 2, anchor: mainrs

    2025-09-11T13:18:44.913Z In(182) vmkernel: cpu82:2100975)pf_rollback_rules: rs_num: 4, anchor: mainrs

    2025-09-11T13:18:44.913Z In(182) vmkernel: cpu82:2100975)pf_rollback_rules: rs_num: 5, anchor: mainrs

    2025-09-11T13:18:44.913Z In(182) vmkernel: cpu82:2100975)pf_rollback_rules: rs_num: 6, anchor: mainrs

    2025-09-11T13:18:45.016Z In(182) vmkernel: cpu82:2100975)pfr_attach_table: nic-#######-eth0-vmware-sfw.2: ERROR ***************** local root table <Address Set UUID> not found

    2025-09-11T13:18:45.016Z In(182) vmkernel: cpu82:2100975)pfioctl: DIOCADDRULE failed with error 22

    2025-09-11T13:18:45.016Z In(182) vmkernel: cpu82:2100975)VSIPConversionCreateRuleSet: Cannot insert ## rule #######: 22

    2025-09-11T13:18:45.016Z In(182) vmkernel: cpu82:2100975)pf_rollback_rules: rs_num: 1, anchor: mainrs

    2025-09-11T13:18:45.016Z In(182) vmkernel: cpu82:2100975)pf_rollback_rules: rs_num: 2, anchor: mainrs

    2025-09-11T13:18:45.016Z In(182) vmkernel: cpu82:2100975)pf_rollback_rules: rs_num: 4, anchor: mainrs

    2025-09-11T13:18:45.016Z In(182) vmkernel: cpu82:2100975)pf_rollback_rules: rs_num: 5, anchor: mainrs

    2025-09-11T13:18:45.016Z In(182) vmkernel: cpu82:2100975)pf_rollback_rules: rs_num: 6, anchor: mainrs

Environment

VMware NSX 4.2.0

VMware NSX 4.2.1.x

Cause

This issue occurs due to a bug where the global_addrset_mode_enabled setting is incorrectly stored as false in Corfu, even though the API call GET /api/v1/infra/settings/firewall/security reports it as true.

When global_addrset_mode_enabled is set to false, each VM creates its own copy of static address sets, leading to multiple duplicate copies on the host. When set to true, a single copy of the global address sets is created on the host and shared by all VMs, preventing duplication.

Starting with NSX 4.2, the only supported value for global_addrset_mode_enabled is true. If a host running NSX 4.2 VIBs receives global_addrset_mode_enabled=false, it will not create any static or global address sets. As a result, any DFW rules that reference address sets will not be programmed on the VMs, causing the DFW policies to show a Failed Realization status.

Resolution

Permanent Fix:

Upgrade to 4.2.2.1 or later release

 

Workaround:

*Performing these workarounds steps should also permanently fix the issue. 

  • Toggle global_addrset_mode_enabled to false and then true again via API:

Step 1: Gather the body required to perform these steps:

GET https://<Manager IP>/policy/api/v1/infra/settings/firewall/security

{

    "idfw_enabled": false,

    "idfw_event_log_scraper_enabled": false,

    "idfw_loginsight_enabled": false,

    "resource_type": "DfwFirewallConfiguration",

    "id": "security",

    "display_name": "security",

    "path": "/infra/settings/firewall/security",

    "relative_path": "security",

    "parent_path": "/infra",

    "remote_path": "",

    "unique_id": "9cd8b89a-f7e3-473d-####-##########",

    "realization_id": "9cd8b89a-f7e3-473d-####-##########",

    "owner_id": "0ea7293f-a909-4a7c-####-##########",

    "origin_site_id": "0ea7293f-a909-4a7c-####-##########",

    "marked_for_delete": false,

    "overridden": false,

    "enable_firewall": true,

    "disable_auto_drafts": false, 

    "global_addrset_mode_enabled": true,   <------------- You will use this attribute to toggle (this value can be 'true' or 'false')

    "_create_time": 1662999213629,

    "_create_user": "system",

    "_last_modified_time": 1663098757027,

    "_last_modified_user": "admin",

    "_system_owned": false,

    "_protection": "NOT_PROTECTED",

    "_revision": 4

}

 

Step 2: Copy API response from Step 1 and use it as the payload. Change "global_addrset_mode_enabled" to false and invoke below API to update settings

PUT https://<Manager IP>/policy/api/v1/infra/settings/firewall/security

{

    "idfw_enabled": false,

    "idfw_event_log_scraper_enabled": false,

    "idfw_loginsight_enabled": false,

    "resource_type": "DfwFirewallConfiguration",

    "id": "security",

    "display_name": "security",

    "path": "/infra/settings/firewall/security",

    "relative_path": "security",

    "parent_path": "/infra",

    "remote_path": "",

    "unique_id": "9cd8b89a-f7e3-473d-####-##########",

    "realization_id": "9cd8b89a-f7e3-473d-####-##########",

    "owner_id": "0ea7293f-a909-4a7c-####-##########",

    "origin_site_id": "0ea7293f-a909-4a7c-####-##########",

    "marked_for_delete": false,

    "overridden": false,

    "enable_firewall": true,

    "disable_auto_drafts": false,     

    "global_addrset_mode_enabled": false,     <--------------------- Toggle this attribute to 'false'   

    "_create_time": 1662999213629,

    "_create_user": "system",

    "_last_modified_time": 1663098757027,

    "_last_modified_user": "admin",

    "_system_owned": false,

    "_protection": "NOT_PROTECTED",

    "_revision": 4

}

 

Step 3: Copy API response from Step 2 and use it again as payload to revert "global_addrset_mode_enabled" back to true.

 

PUT https://<Manager IP>/policy/api/v1/infra/settings/firewall/security

{

    "idfw_enabled": false,

    "idfw_event_log_scraper_enabled": false,

    "idfw_loginsight_enabled": false,

    "resource_type": "DfwFirewallConfiguration",

    "id": "security",

    "display_name": "security",

    "path": "/infra/settings/firewall/security",

    "relative_path": "security",

    "parent_path": "/infra",

    "remote_path": "",

    "unique_id": "9cd8b89a-f7e3-473d-####-##########",

    "realization_id": "9cd8b89a-f7e3-473d-####-##########",

    "owner_id": "0ea7293f-a909-4a7c-####-##########",

    "origin_site_id": "0ea7293f-a909-4a7c-####-##########",

    "marked_for_delete": false,

    "overridden": false,

    "enable_firewall": true,

    "disable_auto_drafts": false,

    "global_addrset_mode_enabled": true,  <--------------------- Toggle this attribute back to 'true' (original value)

    "_create_time": 1662999213629,

    "_create_user": "system",

    "_last_modified_time": 1663098757027,

    "_last_modified_user": "admin",

    "_system_owned": false,

    "_protection": "NOT_PROTECTED",

    "_revision": 5

}

  • Validate that the global_addrset_mode_enabled setting is pushed down to the ESXi hosts
    • As root user on an impacted ESXi host, issue the following command:
      • /opt/vmware/nsx-nestdb/bin/nestdb-cli --beautify --json --cmd get vmware.nsx.nestdb.GlobalConfigMsg | grep global_addrset_mode_enabled

    • The returned value should now be true.

  • Return to the UI and make any DFW rule change. The policies should return to a Success status.
Powered by Wolken Software