nestdb is not running and NSX ports on the ESXi are blocked

Products

• VMware NSX

Issue/Introduction

• NSX ports on an ESXi are blocked for VMs that migrated to the ESXi or that are newly created on the ESXi.
• You see messages like below in nsx-syslog on the ESXi.
  

  <Time Stamp> cfgAgent[2103128]: NSX 2103128 - [nsx@6876 comp="nsx-controller" subcomp="cfgAgent" tid="E06A9700" level="warn"] DaemonHealthMonitor: nestdb echo timeout (60 sec)
<Time Stamp> cfgAgent[2103128]: NSX 2103128 - [nsx@6876 comp="nsx-controller" subcomp="cfgAgent" tid="E06A9700" level="warn"] DaemonHealthMonitor: ping nestdb failed

  <Time Stamp> nsx-proxy[2103523]: NSX 2103523 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="2103548" level="WARNING"] StreamConnection[3064398 Connecting to tcp://127.0.0.1:2480 sid:3064398] Couldn't connect to 'tcp://127.0.0.1:2480' (error: 111-Connection refused)
<Time Stamp> nsx-proxy[2103523]: NSX 2103523 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="2103548" level="WARNING"] StreamConnection[3064398 Error to tcp://127.0.0.1:2480 sid:-1] Error 111-Connection refused
<Time Stamp> nsx-proxy[2103523]: NSX 2103523 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="2103548" level="WARNING"] RpcConnection[3064398 Connecting to tcp://127.0.0.1:2480 0] Couldn't connect to tcp://127.0.0.1:2480 (error: 111-Connection refused)
<Time Stamp> nsx-proxy[2103523]: NSX 2103523 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="2103548" level="WARNING"] RpcTransport[0] Unable to connect to tcp://127.0.0.1:2480: 111-Connection refused
<Time Stamp> nsx-proxy[2103523]: NSX 2103523 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nestdb-client" tid="2103523" level="WARNING"] NestDbClient: failed to get stub to tcp://127.0.0.1:2480, retrying in 5000 ms...

• nestdb is not running on the ESXi.
  To verify nestdb status, run the command on the ESXi.
[root@esxi01:~] /etc/init.d/nsx-nestdb status
NSX-NESTDB is not running


Environment

VMware NSX-T Datacenter 3.x
VMware NSX 4.x

Cause

There is a known issue that nestdb sometimes fails to start on ESXi boot due to a race condition.

NSX configurations on such an ESXi are not effective.

Resolution

Verify nestdb is running after ESXi boot.
/etc/init.d/nsx-nestdb status

If not manually start nestdb and make sure it's running.
[root@esxi:~] /etc/init.d/nsx-nestdb start
NSX-NESTDB started
[root@esxi:~] /etc/init.d/nsx-nestdb status
NSX-NESTDB is running

NSX 4.2 contains a fix for an issue that nestdb fails to start after ESXi boot.