Error: "No space left on device" while running nsx-nestdb service on host transport nodes

Products

• VMware Telco Cloud Platform
• VMware vSphere ESXi

Issue/Introduction

• The control channel to the transport node drops, showing alarms in the user interface.
• When checking the service status on the affected host transport nodes, the nsx-nestdb service is stopped.
• Attempts to start the service manually result in a loop of crashes.

• The following log patterns appear in /var/run/log/nsx-syslog on the host:

leveldb::DB::Write() failed: IO error: /var/lib/vmware/nsx/nestdb/db/######.ldb: No space left on device Encountered IO error, exiting.

• The /var/log/vobd.log file shows related ramdisk errors:

[vob.visorfs.ramdisk.full] Cannot extend visorfs file /var/lib/vmware/nsx/nestdb/db/######.ldb because its ramdisk (nestdb) is full. [esx.problem.visorfs.ramdisk.full] The ramdisk 'nestdb' is full.

• Additionally, a nestdb core dump is generated under /var/core/nestdb-server-zdump.###.

Environment

• TCP: 3, 4.x, 4.0.1, 5.x, 5.0. 5.0.1, 5.0.2
• ESXi: 7.0U2s, 8.0U2b, 8.0U3g

Cause

• The nestdb ramdisk runs out of available space due to an accumulation of database log files.
• This prevents the nsx-nestdb service from performing disk write operations, causing the process to fail and exit immediately.

Resolution

• Generate and download a fresh configuration backup of the NSX Manager cluster before performing any command-line operations on the transport nodes. This is the required first step. For structural execution steps, see the NSX Administration Guide: Back up and Restore the NSX Manager Appliance techdoc.

• Log in to the affected ESXi host via SSH and run the following command to free up the ramdisk space:

  rm -rf /var/lib/vmware/nsx/nestdb/db/*
  

  For deep dive instructions on safe handling of filesystem partitions or tracking files under the nestdb path, reference KB 322542: nestdb agent is down on an NSX prepared ESXi Host

• Start the service manually to allow it to rebuild a clean database instance:

  /etc/init.d/nsx-nestdb start
  

  To cross reference expected logs during a service initialisation phase, review the NSX Command Line Interface Reference Guide: Service Commands techdoc.

• Confirm that the service remains in a running state and does not loop into another crash:

  /etc/init.d/nsx-nestdb status
  

  If you notice recurring failures or status check exceptions during validation, reference the troubleshooting workflows found in KB 410581: Troubleshooting Control Channel To Transport Node Down connection issues].

• If the issue persists or the ramdisk fills immediately after a restart, upgrade to the latest patch release.
• Product downloads can be acquired via your designated support portal.