nsx-nestdb fails to start on vSphere ESX after services.sh restart

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

• Services were restarted on vSphere ESX with services.sh restart
• NSX connected VMs have lost network connectivity
• NSX UI shows status Unknown for the host
• NSX UI shows Tunnels as Not available
• nsxcli -c get controllers  returns  
  % Failed to get controller list
• /etc/init.d/nsx-nestdb status returns
  NSX-NESTDB is not running

• Hosts shell log shows time of services.sh restart
  var/run/log/shell.log
[TIMESTAMP 08:30:00] shell[####]: [root]: services.sh restart

• Hosts syslog shows opsagent stopped at 08:30:10
  var/run/log/syslog
[TIMESTAMP 08:30:10] NSX[####]: Shutting down opsAgent service
[TIMESTAMP 08:30:18] NSX[####]: opsAgent service is stopped

• Host shows start of nestdb nestdb shutdown at 08:30:22
  /var/run/log/syslog
[TIMESTAMP 08:30:22] NSX[####]: Shutting down NSX-NESTDB service
[TIMESTAMP 08:31:17] NSX[####]: NSX-NESTDB service is stopped

• Time between opsagent and nestdb shutdown: 12 seconds. Time to stop nestdb: 55 seconds

Environment

VMware NSX Data Center 3.x

Cause

Nsx-nestdb takes too long to shutdown if nsx-opsagent shutdown takes longer than 5 seconds. The start attempt for nsx-nestdb happens before the service was stopped, preventing it from starting successfully.

Resolution

This issue is resolved in VMware NSX 4.x and above available at Broadcom Downloads
Steps to locate and download Broadcom products and software are available at Download Broadcom products and software 

Workaround:
Manually start nsx-nestdb service with the following command:

/etc/init.d/nsx-nestdb start