NSX Manager cluster expansion fails due to /var/log partition exhaustion

Products

• VMware NSX

Issue/Introduction

When adding new NSX Manager nodes to a cluster, the deployment may fail or new nodes may stay in an UNKNOWN or FAILED state. This occurs when the /var/log partition on existing or newly deployed nodes reaches 100% capacity, preventing the Cluster Boot Manager (CBM) from synchronizing the database.

Symptoms:

• Adding a second or third NSX Manager node fails.
• Command get cluster status shows newly added nodes in UNKNOWN or FAILED state.
• Deployment logs indicate storage or partition exhaustion.
• Unable to login using root

Environment

VMware NSX-T Data Center 3.1

Cause

The /var/log partition on the surviving manager or newly deployed OVA nodes reaches 100% capacity due to Dropwizard metrics (.csv files). This prevents the Cluster Boot Manager (CBM) from completing the synchronization of the Corfu database.

Resolution

To resolve this issue, perform the following steps:

1. Clear Disk Space: Follow the cleanup steps in KB 319977 to remove large .csv files and modify the corfu-server-wrapper.conf to disable metrics.

2. Deactivate Cluster (If Quorum is Lost): If the cluster is in a degraded state and cannot accept new nodes, run the following command on the healthy node to reset it to a standalone state:

    

   deactivate cluster

   Note: Ensure the surviving node is fully synced by checking /config/corfu/LAYOUT_CURRENT.ds before running this.

3. Manual Node Join: If UI deployment continues to fail, deploy the node via OVA and join it manually using the CLI:

    

   join <Primary-Manager-IP> cluster-id <Cluster-UUID> username admin password <Admin-Password> thumbprint <SHA-256-Thumbprint>