VMs on NSX prepared hosts lost networking connectivity

Products

• VMware NSX

Issue/Introduction

NSX ESXi host logs have the following symptoms:

 

1.  T1 DR next hop mac address is  00:00:00:00:00:00

_{net-vdr -N -l 68513eba-####-####-####-5526a2a894ebDR 68513eba-8a49-4362-896d-5526a2a894eb ARP Information :
Legend: [S: Static], [V: Valid], [P: Proxy], [I: Interface]
Legend: [N: Nascent], [L: Local], [D: Deleted], [K: linKlif]
Legend: [A: AutoRefresh]Network           Mac                  Flags      Expiry     Uptime     SrcPort    Refcnt     Lif UUID
-------           ---                  -----      ------     ------     -------    ------     ---------
169.254.0.2       00:00:00:00:00:00    N          0          3          0          2          3cabff5c-####-####-####-a7bae5a7cea1} 

2. Host nestdb connection is down

_{2024-09-10T08:32:15Z nsx-sha: NSX 232216507 - [nsx@6876 comp="nsx-esx" subcomp="nsx-sha" username="root" level="WARNING" s2comp="nestdb-client"] Exception in connecting nestdb, will retry in 10s. Exception:[Errno 111] Connection refused, trace:Traceback (most recent call last):   File "/usr/lib/vmware/netopa/lib/python/sha/core/profile/_nestdb_client.py", line 172, in _setup_monitor     self._stub = NsxRpcClient(NestDb_Stub, self._endpoint)   File "/usr/lib/vmware/nsx-proto2/python/vmware/nsx/rpc/client/client.py", line 64, in __init__     self.own_connection.Connect(endpoint, sec_ctx=sec_ctx, retry_policy=retry_policy)   File "/usr/lib/vmware/nsx-proto2/python/vmware/nsx/rpc/client/transport.py", line 104, in Connect     self._Connect(endpoint, sec_ctx)   File "/usr/lib/vmware/nsx-proto2/python/vmware/nsx/rpc/client/transport.py", line 117, in _Connect     self.sock.connect((addr, port))   File "/lib64/python3.8/gevent/_socket3.py", line 335, in connect     raise error(err, strerror(err)) ConnectionRefusedError: [Errno 111]}

 

Environment

NSX 3.x

NSX 4.x

Cause

The issue was caused by the host nestdb daemon mulfuction. 

Resolution

Please open a support case to find the root cause of the issue before trying the following workaround:

 

Reboot the affected ESXi host. 

Or issue the following command on the host: 

/etc/init.d/nsx-nestdb restart