NSX Management Plane Service Disruptions and Corfu Network Exceptions Driven by ArpTable Synchronization Mechanics and Overlapping NAT Subnets

Products

• VMware NSX

Issue/Introduction

In VMware NSX, core management services (proton, nsx-cbm, ccp) and the user interface may crash or restart unexpectedly.

Observed symptoms and error strings include:

• The NSX Manager UI displays: Some appliance components are not functioning properly.
  
  
• Component health status reports SEARCH:DOWN and MANAGER:DOWN with Error code: 101.
  
  
• Running get cluster status on the NSX Manager CLI returns error messages or lists services as DOWN.
  
  
• The following logs (/var/log/corfu/tanuki.log, /var/log/corfu/corfu-compactor-audit.log, or /var/log/proton/proton-tomcat-wrapper.log) display compactor failures and JVM memory exhaustion errors, specifically: "The JVM has run out of memory" and "java.lang.OutOfMemoryError: Java heap space". 
  
  
• The NSX Manager nodes experience severe resource exhaustion, resulting in the generation of critical out-of-memory (OOM) heap dumps (such as proton_oom.hprof, cbm_oom.hprof, and ccp_oom.hprof) located in the /image/core/ directory.
  
  
• /var/log/proton/nsxapi.log displays frequent Corfu network disconnection exceptions (org.corfudb.runtime.exceptions.NetworkException: Disconnected) on port 9000 during write/read log requests.
  
  
• A review of the /controller/adaptor-ufo/adaptor_ufo_dump file reveals that the ARP entry count exceeds 50,000.
  
  

  nsx_manager# grep -w arp_entry_type adaptor_ufo_dump | wc -l  
  50000+
  
  OR  
  
  nsx_manager# grep -w entity_id  adaptor_ufo_dump | wc -l
  50000+
  
  

Environment

VMware NSX

Cause

This issue is caused by a management plane out-of-memory (OOM) condition resulting from excessive database synchronization and trigger processing during high-scale configuration events or Proton service restarts.

The system inherently writes ArpTable records to the Corfu database on every ARP proxy L3 trigger pass, regardless of actual data changes, while sequential L3 Triggers for identical logical routers queue and execute independently.

In dense topologies where Tier-1 SNAT rules overlap with Tier-0 uplink subnets, this concurrent processing model generates a high volume of table rewrites and handler executions that exceeds the maximum limit of 50,000 ARP entries per Logical Router.

This resulting data throughput rapidly expands the transaction log and Corfu checkpoint sizes, which eventually outstrips the allocated heap capacity of the database compactor and causes the service to crash (generating .hprof dumps).

Resolution

Resolution

A permanent fix for this issue will be introduced in a future release of NSX. This issue currently affects versions up to NSX 4.2.4 and NSX 9.1. Please click Subscribe on this knowledge base article to be notified when the official fix becomes available: Subscribe to a Broadcom knowledge article.

Workaround

To mitigate this issue in the interim, please engage Broadcom Support for targeted remediation:

• File a ticket with Broadcom Support.

• A Support Engineer will assist you in generating and extracting a complete Corfu database dump from the affected NSX Manager.

• Broadcom Support will analyze the database dump to isolate the affected tables and generate a targeted, manual cleanup plan specific to your environment.

 

Additional Information

NSX is Impacted by JDK-8330017: ForkJoinPool Stops Executing Tasks.

Cleanup Stale from the Corfu database.

Edge global ARP table usage high alarm.

Cluster instability due to Proton (Manager) service Out Of Memory.