VMs skipping DFW rules due to IPs missing from addrsets on ESX hosts, causing traffic to be dropped due to DFW filtering

Products

• VMware NSX
• VMware vDefend Firewall

Issue/Introduction

• Traffic from Workload VM stops working.
• Allow Rule intended for the Workload VM doesn't get Hit.
• Default Block Rule if configured drops the traffic.

The following log will be present in cloudnet ccp logs:

ERROR pool-16-thread-3 InternalDatastoreImpl 17912 - [nsx@6876 comp="nsx-controller" errorCode="CCP00000001" level="ERROR" subcomp="replication"] Unknown exception caught
com.vmware.nsx.platform.service.ServiceException: org.corfudb.runtime.exceptions.unrecoverable.UnrecoverableCorfuError: Unexpected exception during commit
XXXXXXXXX
Caused by: java.lang.IllegalArgumentException: Serialized Value is too big (3261255).
        at com.vmware.nsx.platform.kvstore.adapter.corfudb.ObjectViewSerializer.serializeDatum(ObjectViewSerializer.kt:139) ~[nsx_ccp_distribution_deploy.jar:?]

The following log  will be present in NSX Manager syslog:

NSX 17912 - [nsx@6876 comp="nsx-controller" level="WARNING" subcomp="transport-node-adapter"] To be deleted value ip {#012 ipv4: x.x.x.x#012}#012mac {#012 mac: xx:xx:xx:xx:xx:xx#012}#012 doesn't match old value ip {#012 ipv4: x.x.x.x#012}#012mac {#012 mac: xx:xx:xx:xx:xx:xx#012}#012updated_time: 1111111111#012

NSX 17912 - [nsx@6876 comp="nsx-controller" level="WARNING" subcomp="transport-node-adapter"] To be deleted value ip {#012 ipv4: x.x.x.x#012}#012mac {#012 mac: xx:xx:xx:xx:xx:xx#012}#012 doesn't match old value ip {#012 ipv4: x.x.x.x#012}#012mac {#012 mac: xx:xx:xx:xx:xx:xx#012}#012updated_time: 111111111#012

Environment

VMware NSX 3.1.x

Cause

There is an issue with the NSX DB update logic within the ESXi host that results in this issue.

Resolution

Restart the Controller service on all 3 Managers as admin user
     > restart service controller

 

 

Additional Information

Issue resolved in 3.1.3.2, 3.1.4.0 (VMC M16), and 3.2.0 (VMC M17).