SDDC Manager password remediation for NSX-T Edge nodes failing with Error: Unable to get transport nodes from NSX-T cluster

Products

• VMware Cloud Foundation

Issue/Introduction

To perform list of troubleshooting to resolve the error with password remediation and allow successful credential remediate operation in Password Management tab of the SDDC Manager UI.

Symptoms:
Password remediate for NSX-T Edge node failing with below error in SDDC UI: 

Unable to obtain SSH connectivity to entity: <nsx-t edge node fqdn>



SDDC Manager > operationsmanager.log 



 

 

Environment

Vmware Cloud Foundation 4.5
VMware Cloud Foundation 4.x

Cause

Mismatched hostname of Edge Node in SDDC Manager Platform DB

Resolution

• 1. ssh to sddc manager with vcf user and su to root
  2. Run the below command on SDDC Manager to fetch all the edge nodes deployed 

     • curl -X GET -k -s -w "\n" -u 'admin:<password>' https://<nsxt-manager-fqdn>/api/v1/transport-nodes | json_pp

     • Note: Replace the nsxt-manager fqdn and the password in the curl command
  3. From the response of above curl command, pick the id of the edge node
  4. Run the below command to list the details of the edge node for which password remediate is failing

     • curl -X GET -k -s -w "\n" -u 'admin:<password>' https://<nsxt-manager-fqdn>/api/v1/transport-nodes/<id> | json_pp

     • Note: Replace the <id> with the edge node id gathered in Step 3.
  5. From the above response check the hostname value under node_deployment_info -> node_settings -> hostname
  6. The hostname here should exactly match the hostname on SDDC DB & in the NSXT Edge CLI ---> Match FQDN / Short name / Lower or upper case.
     
     
     • Update nsxt_edge_nodes in nsxt_edge_cluster table with the matching hostname in SDDC Platform DB - Take snapshot of SDDC Manager before making any changes to SDDC DB
       # psql -h localhost -U postgres -d platform
       
       # \x (To Expand display)
       
       # select id,status,name,nsxt_edge_nodes,source_id from nsxt_edge_cluster;     
       
       Example Output:
       
       id | xxxxxxx-xxxx-xxxx-xxx-xxxxxxxxxxx -------> Note the id of the nsxt_edge_cluster which needs to be updated
       creation_time | 1736818011178
       modification_time | 1736820241138
       status | ACTIVE
       name | ab-bcd-def01
       nsxt_edge_nodes | [ {"vmManagement IpAddress": "##.##.##.##", "vmHostname": "<nsxedge-FQDN>", "sourceId": "######.####.####.###########", "id
       ": "######.####.####.###########"}, {"vmManagement IpAddress": "##.##.##.##", "vmHostname": "nsxedge-FQDN", "sourceId": "######.####.####.###########", "id": "######.####.####.###########"}]
       
       # update nsxt_edge_cluster set nsxt_edge_nodes='<above output highlighted in blue>' where id='<nsxt_edge_cluster id noted from above step>';
       
       Example:
       update nsxt_edge_cluster set nsxt_edge_nodes='[ {"vmManagement IpAddress": "##.##.##.##", "vmHostname": "<nsxedge-FQDN>", "sourceId": "######.####.####.###########", "id
       ": "######.####.####.###########"}, {"vmManagement IpAddress": "##.##.##.##", "vmHostname": "nsxedge-FQDN", "sourceId": "######.####.####.###########", "id": "######.####.####.###########"}]' where id='######.####.####.###########';
       
       
  7. Refresh the UI in sddc manager, retry password remediation for NSX-T Edge nodes.
  8. Once successful please reset the db entries back to the original values as it was. This is to make sure none of the other workflows would break.