Shrinking NSX Edge Cluster stuck /failed at NSX Tier-O BGP Neighbor Configuration stage

Products

• VMware SDDC Manager / VCF Installer
• VMware NSX

Issue/Introduction

• Shrinking task stuck at SDDC Manager for NSX edge node cluster and not progressing anywhere -
  
  
  
  
  
• Failed task details - 
  

  Delete NSX Tier-0 BGP Neighbor Configuration - Failed
  Description - Delete NSX Tier-0 BGP Neighbor Configuration
  
  Progress Messages  - Pre validate of Tier0 BGP neighbor config deletion failed
  
  Error
  Message: Pre validate of Tier0 BGP neighbor config deletion failed
  Remediation Message:
  Reference Token: #######
  Cause: neighbor description(s) are required

  
  
  
  
• Even deleting its associative task using its ID, restarting SDDC services or even rebooting the SDDC appliance will not make any change and the Shrinking task keeps on moving. 

Environment

SDDC Manager 9.0
NSX 4.x 

Cause

The issue is caused when the edge cluster is deployed via the API.
This method updates the SDDC Manager database with an incomplete status, leaving the cluster entry in the "ACTIVATING" or "SHRINKING "state.

Verify the same through SDDC DB using the following commands, where s stale entry is showing with either "ACTIVATING" or "SHRINKING" status in the nsxt_edge_cluster table-

1. psql -h localhost -U postgres
2. \c platform
3. platform # select * from nsxt_edge cluster;

Example - 

Resolution

To fix this issue, change the SHRINKING status of the NSX-T cluster to ACTIVE state in the SDDC database by tracking its associated ID using the following steps - 

Note - please take a snapshot (backup point) before proceeding with the below steps. 


1. login to SDDC database platform  

psql -h localhost -U postgres
\c platform

2. Find the NSXT Edge cluster table and update status from "SHINKING" or "ACTIVATING" to "ACTIVE"

platform # select * from nsxt_edge cluster;

platform=# update nsxt_edge_cluster set status='ACTIVE' where id='<id>';
UPDATE 1

3. Check and confirm active status -

platform=# select * from nsxt_edge_cluster;
platform=# \q

 

4. Post that the SHIRNKING status for the NSX Edge cluster becomes ACTIVE and grayed out options are available to proceed further -