nestdb is not running and NSX ports on the ESXi are blocked
search cancel

nestdb is not running and NSX ports on the ESXi are blocked

book

Article ID: 376731

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • NSX ports on an ESXi is blocked for VMs that migrated to the ESXi or that are newly created on the ESXi.
  • You see messages like below in nsx-syslog on the ESXi.

    <Time Stamp> cfgAgent[2103128]: NSX 2103128 - [nsx@6876 comp="nsx-controller" subcomp="cfgAgent" tid="E06A9700" level="warn"] DaemonHealthMonitor: nestdb echo timeout (60 sec)
    <Time Stamp> cfgAgent[2103128]: NSX 2103128 - [nsx@6876 comp="nsx-controller" subcomp="cfgAgent" tid="E06A9700" level="warn"] DaemonHealthMonitor: ping nestdb failed

    <Time Stamp> nsx-proxy[2103523]: NSX 2103523 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="2103548" level="WARNING"] StreamConnection[3064398 Connecting to tcp://127.0.0.1:2480 sid:3064398] Couldn't connect to 'tcp://127.0.0.1:2480' (error: 111-Connection refused)
    <Time Stamp> nsx-proxy[2103523]: NSX 2103523 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="2103548" level="WARNING"] StreamConnection[3064398 Error to tcp://127.0.0.1:2480 sid:-1] Error 111-Connection refused
    <Time Stamp> nsx-proxy[2103523]: NSX 2103523 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="2103548" level="WARNING"] RpcConnection[3064398 Connecting to tcp://127.0.0.1:2480 0] Couldn't connect to tcp://127.0.0.1:2480 (error: 111-Connection refused)
    <Time Stamp> nsx-proxy[2103523]: NSX 2103523 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="2103548" level="WARNING"] RpcTransport[0] Unable to connect to tcp://127.0.0.1:2480: 111-Connection refused
    <Time Stamp> nsx-proxy[2103523]: NSX 2103523 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nestdb-client" tid="2103523" level="WARNING"] NestDbClient: failed to get stub to tcp://127.0.0.1:2480, retrying in 5000 ms...

  • nestdb is not running on the ESXi.
    To verify nestdb status, run the command on the ESXi.
    [root@esxi01:~] /etc/init.d/nsx-nestdb status
    NSX-NESTDB is not running

Environment

VMware NSX-T Datacenter 3.x
VMware NSX 4.x

Cause

There is a known issue that nestdb sometimes fails to start on ESXi boot due to a race condition.

NSX configurations on such an ESXi are not effective.

Resolution

Verify nestdb is running after ESXi boot.
/etc/init.d/nsx-nestdb status

If not manually start nestdb and make sure it's running.
[root@esxi:~] /etc/init.d/nsx-nestdb start
NSX-NESTDB started
[root@esxi:~] /etc/init.d/nsx-nestdb status
NSX-NESTDB is running

NSX 4.2 contains a fix for an issue that nestdb fails to start after ESXi boot.