vMotioned VMs are not Reachable Due to Failed DVFilter State Restoration while the host has NESTDB service Down.

Products

• VMware vDefend Firewall

Issue/Introduction

vMotioned VMs are not reachable over the network following the NESTDB being down on the hosts.

Symptoms:

• The host is rebooted, and vMotion was initiated towards the available hosts.
• Upon tracing events for the VMs, it is observed that the VM ports went into a blocked state due to failed DVFilter state restoration: Bringing down port due to failed DVFilter state restoration and failPolicy of FAIL_CLOSED.
• The NESTDB service on the source host was down even before the Filters Export was started, and it stayed down until NESTDB was manually restarted.

 vmkernel logs from both the source and destination hosts provide evidence of the issue:

Export logs on source host (0 tables, 31 rules):

2025-03-23T15:36:24.729Z In(182) vmkernel: cpu74:2143965)nic-########-eth0-vmware-sfw.2, Version 1100
2025-03-23T15:36:24.730Z In(182) vmkernel: cpu74:2143965)GetSavedStateLenTLV total: 64030, srchost: 52/36/1, tables: 24/8/0, rules: 9835/9819/3
2025-03-23T15:36:24.730Z In(182) vmkernel: cpu74:2143965) states: 24/8/0, sidcache: 24/8/0, attrs: 24/8/0, algs: 564/548/20, algports: 24/8/0
2025-03-23T15:36:24.730Z In(182) vmkernel: cpu74:2143965) fqdn_vm_node_prop: 31/15/3, fqdn: 24/8/0, miscs: 2164/2148/5, x:51200
2025-03-23T15:36:24.730Z In(182) vmkernel: cpu74:2143965) sip_persist: 24/8/0
2025-03-23T15:36:24.730Z In(182) vmkernel: cpu74:2143965)Sending length of 64094

025-03-23T15:36:24.730Z In(182) vmkernel: cpu74:2143965)Exporting nic-########-eth0-vmware-sfw.2, Version 1100
2025-03-23T15:36:24.730Z In(182) vmkernel: cpu74:2143965)ExportStateTLV total: 12830, srchost: 52/36/1, tables: 24/8/0, rules: 9835/9819/31
2025-03-23T15:36:24.730Z In(182) vmkernel: cpu74:2143965) states: 24/8/0, sidcache: 24/8/0, attrs: 24/8/0, algs: 564/548/20, algports: 24/8/0
2025-03-23T15:36:24.730Z In(182) vmkernel: cpu74:2143965) fqdn_vm_node_prop: 31/15/3, fqdn: 24/8/0, miscs: 2164/2148/5
2025-03-23T15:36:24.730Z In(182) vmkernel: cpu74:2143965) sip_persist: 24/8/0

 

Import logs on destination host (0 tables, 31 rules):

2025-03-23T15:36:24.767Z In(182) vmkernel: cpu167:2098510)Importing nic-########-eth0-vmware-sfw.2, Version 1100
2025-03-23T15:36:24.767Z In(182) vmkernel: cpu167:2098510)VSIP module ioctls: disabled
2025-03-23T15:36:24.767Z In(182) vmkernel: cpu167:2098510)ImportStateTLV entry type 12, len 52, cnt 1
2025-03-23T15:36:24.767Z In(182) vmkernel: cpu167:2098510)Importing from source version RELEASEbuild-24302014
2025-03-23T15:36:24.767Z In(182) vmkernel: cpu167:2098510)ImportStateTLV entry type 1, len 24, cnt 0
2025-03-23T15:36:24.767Z In(182) vmkernel: cpu167:2098510)ImportStateTLV entry type 2, len 9835, cnt 31
2025-03-23T15:36:24.767Z In(182) vmkernel: cpu167:2098510)configured filter nic-#######-eth0-vmware-sfw.2
2025-03-23T15:36:24.767Z In(182) vmkernel: cpu167:2098510)filter nic-########-eth0-vmware-sfw.2 flushing flow cache
2025-03-23T15:36:24.767Z In(182) vmkernel: cpu167:2098510)pfr_attach_table: nic-########-eth0-vmware-sfw.2: ERROR ***************** local root table local root table <uuid> not found
2025-03-23T15:36:24.767Z In(182) vmkernel: cpu167:2098510)pfioctl: DIOCADDRULE failed with error 22

Environment

VMware NSX-T Data Center

Cause

• All failures were the same, e.g. Table "local root table <uuid>" not found in import blob.
• This is identified to be a problem from the export side during the blob creation and this is caused by the nestdb service being down during the export. Including 0 tables in the export blob is unexpected behavior. These filters were originally created with 23 tables & 31 rules during their import.
• The nestdb service staying down following the host reboot is a known issue with NSX 4.2.1, see: nsx-nestdb fails to start automatically during ESXi host boot/reboot

Resolution

Workaround:

Disconnect and re-connect the VM's network interface.

Resolution:

Fixed in 4.2.2 and later.