NSX-T DFW Configuration Realization Delay due to NestDB Timeouts

Products

• VMware vDefend Firewall

Issue/Introduction

NSX-T DFW Configuration Realization Delay due to NestDB Timeouts.

 

Symptoms:

• A noticeable delay of 10 to 25 minutes occurs between publishing a policy change in NSX Manager and its actual realization on the ESXi host.
• The issue usually triggers during large configuration updates, such as group membership changes involving a high count of IP addresses.
• Heavy transactions occur between the Central Control Plane (CCP) and Local Control Plane (LCP).
• The nsx-nestdb service may temporarily report a DOWN status or experience RPC timeouts.

 

Log example:

 

From: /var/run/log/nsx-syslog.log

 

1. A large configuration transaction is received by NestDB from the Central Control Plane (CCP)

Example log snippet:

nestdb-server[2103922]: Modify: TransactionID='6075808' Client ID=ccp Size=0.99MB LogTx?=1

nestdb-server[2103922]: Modify: TransactionID='6075808' Client ID=ccp Size=1.05MB LogTx?=1

2. Immediately afterward, NestDB RPC calls fail due to resource constraints, causing timeouts

nsx-sha: NSX 2104803 - [nsx@6876 comp="nsx-esx" subcomp="nsx-sha" username="root" level="INFO" s2comp="metric-nsx.daemon.daemon-health"] Error in echo Daemon:nsx-nestdb, error: RPC Call Failed with CallStatus(code=TIMEOUT), trace:Traceback (most recent call last)
\

3. The RPC connection closes due to a network error/timeout condition

nestdb-server[2103922]: NSX 2103922 - [nsx@6876 comp="nsx-esx" subcomp="nsx-nestdb" s2comp="nsx-rpc" tid="2103933" level="INFO"] RpcConnection[226 Connected on tcp://0.0.0.0:24800] Closing (network error)
nestdb-server[2103922]: NSX 2103922 - [nsx@6876 comp="nsx-esx" subcomp="nsx-nestdb" s2comp="nsx-rpc" tid="2103933" level="INFO"] RpcConnection[226 Closed on tcp://0.0.0.0:24800] Notifying channels on connection down (network error)

4. The Supportability and Health Assistant (SHA) daemon marks the service state as DOWN

nsx-sha: NSX 2104803 - [nsx@6876 comp="nsx-esx" subcomp="nsx-sha" username="root" level="INFO"] [name:daemon_health_monitor][id:217c1efc-c0b1-41ca-a65a-e783f7303b6f] agent: 3, state changes from {'state': 1, 'last_update_time': 20876810} to type: NSX_NESTDB status: DOWN

 

 

---

Environment

NSX-T Data Center 

 

Cause

This issue occurs under heavy transaction loads, such as large IP address churn within groups, depending on the environment. During these times, the nsx-nestdb process becomes too busy to respond to health check requests from the Supportability and Health Assistant (SHA).

Because the process is occupied, the SHA daemon experiences an RPC timeout and changes the nsx-nestdb status to DOWN. The service recovers automatically once heavy processing finishes and realization completes on the datapath.

Resolution

Always limit the configuration scale below the thresholds mentioned in the VMware Configuration Maximums.