VCD Firewall Fails: I/O error on GET request to NSX Manager API endpoint
search cancel

VCD Firewall Fails: I/O error on GET request to NSX Manager API endpoint

book

Article ID: 439141

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • When trying to configure Firewall rules from VCD you see error below:
    “I/O error on GET request to the NSX-T Manager API endpoint — ‘No route to host (Host unreachable)’.”
  • All three NSX Manager nodes are down and core services (Controller/Manager/Https) fail to start.

> get cluster status

  • The /config partition is at 51% utilization and growing, causing nodes to enter read-only mode.

 

  • The management cluster is stuck in DB Syncing and the UI is inaccessible.
  • API calls time out, preventing log bundle generation via CLI.
  • Syslog displays a core dump generated by compactor_oom.
  • The corfu-compactor-audit.log displays "ignoring repeated trim".
    grep prefixTrim corfu-compactor-audit.log
  • Increased the memory to 12gb and run manual compact but this does not complete.

Environment

VMware NSX 4.1.2.4

 

Cause

The automated Corfu database compaction process fails, resulting in an Out-Of-Memory (OOM) condition that causes database bloat, exhausts the /config partition, and crashes management services across all nodes.

Resolution

Workaround:

  1. Restore the primary NSX Manager (VIP node) from a valid backup. Ensure the passcode is available.

  2. Power off the remaining two corrupted NSX Manager nodes.

  3. Verify the restoration is successful and confirm the /config partition usage has returned to a healthy state (e.g., ~1%) by running df -h.
    Note: Acknowledge any UI prompts recommending a three-node manager cluster.

  4. Deploy two new NSX Manager OVA VMs using the same IP addresses as the powered-off nodes (or deploy via the UI). Keep them powered off until the primary node is fully restored and stable.

  5. Once the primary node is stable, power on the newly deployed NSX Manager VMs.

  6. SSH into each new manager and join them to the cluster one at a time using the following command (run as admin):
    join <Good-NSX-Node-IP> cluster-id <cluster-id> thumbprint <Good-NSX-Node-IP-thumbprint> username admin password <admin-password>

  7. After all nodes are joined, verify that all services are in an UP state and that the cluster status is STABLE.

  8. Delete previously corrupted NSX Manager VM's from VCenter.

Additional Information

Restore NSX Manager from Backup

Powered by Wolken Software