nestdb is not running and NSX ports on the ESXi are blocked

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

• NSX ports on an ESXi are blocked for VMs that migrated to the ESXi or that are newly created on the ESXi.
• You see messages like below in /var/run/log/nsx-syslog.log on the ESXi.
  

  <Time Stamp> cfgAgent[2103128]: NSX 2103128 - [nsx@6876 comp="nsx-controller" subcomp="cfgAgent" tid="E06A9700" level="warn"] DaemonHealthMonitor: nestdb echo timeout (60 sec)
<Time Stamp> cfgAgent[2103128]: NSX 2103128 - [nsx@6876 comp="nsx-controller" subcomp="cfgAgent" tid="E06A9700" level="warn"] DaemonHealthMonitor: ping nestdb failed

  <Time Stamp> nsx-proxy[2103523]: NSX 2103523 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="2103548" level="WARNING"] StreamConnection[3064398 Connecting to tcp://127.0.0.1:2480 sid:3064398] Couldn't connect to 'tcp://127.0.0.1:2480' (error: 111-Connection refused)
<Time Stamp> nsx-proxy[2103523]: NSX 2103523 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="2103548" level="WARNING"] StreamConnection[3064398 Error to tcp://127.0.0.1:2480 sid:-1] Error 111-Connection refused
<Time Stamp> nsx-proxy[2103523]: NSX 2103523 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="2103548" level="WARNING"] RpcConnection[3064398 Connecting to tcp://127.0.0.1:2480 0] Couldn't connect to tcp://127.0.0.1:2480 (error: 111-Connection refused)
<Time Stamp> nsx-proxy[2103523]: NSX 2103523 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="2103548" level="WARNING"] RpcTransport[0] Unable to connect to tcp://127.0.0.1:2480: 111-Connection refused
<Time Stamp> nsx-proxy[2103523]: NSX 2103523 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nestdb-client" tid="2103523" level="WARNING"] NestDbClient: failed to get stub to tcp://127.0.0.1:2480, retrying in 5000 ms...

• nestdb is not running on the ESXi.
  • To verify nestdb status, run the command on the ESXi.
    • /etc/init.d/nsx-nestdb status
  • Status returns the following
    • NSX-NESTDB is not running


Environment

VMware NSX-T Data Center
VMware NSX

Cause

There is a known issue that service nsx-nestdb sometimes fails to start on ESXi boot due to a race condition.

NSX configurations on ESXi are not affected.

Resolution

• Verify nestdb is running after ESXi boot.
  • /etc/init.d/nsx-nestdb status

• If service is not running, manually start nestdb and make sure it's running:
  • /etc/init.d/nsx-nestdb start
    • NSX-NESTDB started
  • /etc/init.d/nsx-nestdb status
    • NSX-NESTDB is running

NSX 4.2.1.1 express patch contains a fix for this issue.