In NSX-T, IPs are missing from address sets on ESXi hosts, causing traffic to be dropped due to DFW filtering

Products

• VMware NSX

Issue/Introduction

• Traffic for workload VMs experience interruption.
• Allow Rules configured for the workload VMs no longer appear to pass traffic.
• The default Block Rule if configured drops the traffic.
• On the NSX-T Manager, entries similar to the below can be seen in var/log/cloudnet/nsx-ccp.log

ERROR pool-16-thread-3 InternalDatastoreImpl 17912 - [nsx@6876 comp="nsx-controller" errorCode="CCP00000001" level="ERROR" subcomp="replication"] Unknown exception caught
com.vmware.nsx.platform.service.ServiceException: org.corfudb.runtime.exceptions.unrecoverable.UnrecoverableCorfuError: Unexpected exception during commit
XXXXXXXXX
Caused by: java.lang.IllegalArgumentException: Serialized Value is too big (3261255).
        at com.vmware.nsx.platform.kvstore.adapter.corfudb.ObjectViewSerializer.serializeDatum(ObjectViewSerializer.kt:139) ~[nsx_ccp_distribution_deploy.jar:?]

• On the NSX-T Manager, the following entries may also be observed in var/log/syslog

NSX 17912 - [nsx@6876 comp="nsx-controller" level="WARNING" subcomp="transport-node-adapter"] To be deleted value ip {#012 ipv4: <ip-addr-1>#012}#012mac {#012 mac: <mac-addr-1>#012}#012 doesn't match old value ip {#012 ipv4: <ip-addr-2>#012}#012mac {#012 mac: <mac-addr-2>#012}#012updated_time: <epoch-time>#012

Environment

VMware NSX-T Data Center

Cause

This issue can be encountered with a NestDB update where there is a full sync and no preceding delta sync has occurred.

Resolution

This issue is resolved in VMware NSX-T Data Center 3.1.3.3 and 3.2, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround
It is possible to work around this issue by restarting the controllers one by one.