SDDC Manager password remediation for NSX-T Edge nodes failing with Error: Unable to get transport nodes from NSX-T cluster
search cancel

SDDC Manager password remediation for NSX-T Edge nodes failing with Error: Unable to get transport nodes from NSX-T cluster

book

Article ID: 316073

calendar_today

Updated On:

Products

VMware Cloud Foundation VMware SDDC Manager

Issue/Introduction

  • Password remediate for NSX-T Edge node failing with below error in SDDC UI:  
    Unable to obtain SSH connectivity to entity: edge01.example.com

     

    Unable to obtain SSH connection to entity: edge01.example.com
Remediation Message: Please verify that the account is active and is not locked. If the password of the account has expired, manually reset the password in the product and then perform a REMEDIATE operation in the SDDC Manager, to update its stored copy of the password.
Reference Token: T####V
Cause: Unable to get NSX-T Transport Nodes from NSX-T Cluster

     

  • Error in SDDC Manager /var/log/vmware/vcf/operationsmanager/operationsmanager.log 
    ERROR [vef_om, ae##########41, 2aef] [c.v.v.p.u.changers. NsxtEdgeChanger, om-exec-27] Exception occurred while testing NSXT Edge node credentials
com.vmware.vcf.passwordmanager.exception. PasawordUpdateException: Unable to get NSX-T Transport Nodes from NSX-T Cluster
		at com.vmware.vcf.pasawordmanager.helper.NaxtApiUtil.getNaxtTransportNodeByFqdn (NaxtApiUtil.java: 343)
		at com.vmware.vcf.passwordmanager. update. changers.NsxtEdgeChanger.doTest (NsxtEdgeChanger.java: 89)
		at com.vmware.vcf.pasawordmanager.update.changers.AbstractPasswordChanger.updateAsync (AbstractPasswordChanger.java: 430)
		at com.vmware.vcf.passwordmanager. update.changers.AbstractPasswordChanger.doUpdate (AbstractPasswordChanger.java: 199)
		at com.vmware.vcf.passwordmanager. rotate.AbstractPasswordTransactionExecutor$1.call(AbstractPasswordTransactionExecutor. java: 100)
		at com.vmware.vcf.passwordmanager.rotate.AbstractPasswordTransactionExecutor$1.call (AbstractPasswordTransactionExecutor. java: 88)
		at org.springframework.cloud. sleuth. instrument.async.TraceCallable.call (TraceCallable.java: 71)
		at java.base/java.util.concurrent. FutureTask. run (FutureTask. java:264)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java: 1128)
		at java.base/java.util.concurrent. ThreadPoolExecutortWorker.run (ThreadPoolExecutor.java: 628)
		at java.base/java.lang.Thread. run (Thread.java:829)
ERROR [vcf_om, ae##########41, 2aef] [c.v.v.p.u.c.AbstractPasswordChanger, om-exec-27] Unable to get NSX-T Transport Nodes from NSX-T Cluster
com.vmware.vcf.passwordmanager.exception.PasswordUpdateException: Unable to get NSX-T Transport Nodes from NSX-T Cluster
		at com.vmware.vcf.passwordmanager.update.changers.NsxtEdgeChanger.doTest (NsxtEdgeChanger. java:168)
		at com.vmware.vcf.pasawordmanager. update. changers.AbstractPasswordChanger.updateAsync (AbatractPasswordChanger.java:430)
		at com.vmware.vcf.passwordmanager. update.changers. AbstractPasswordChanger.doUpdate (AbstractPasswordChanger.java:199)
		at com.vmware.vcf.passwordmanager.rotate.AbstractPasswordTransactionExecutor$1.call (AbstractPasswordTransactionExecutor. java:100)
		at com.vmware.vcf.pasawordmanager.rotate.AbstractPasswordTransactionExecutor$1.call (AbatractPasawordTransactionExecutor. java: 88)
		at org.springframework.cloud. sleuth. instrument.async.TraceCallable.call (TraceCallable.java: 71)
		at java.base/java.util.concurrent. FutureTask. run (FutureTask. java:264)
		at java.base/java.util.concurrent. ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java: 1128)
		at java.base/java.util.concurrent.ThreadPoolExecutortWorker.run (ThreadPoolExecutor.java: 628)
		at java.base/java.lang. Thread.run (Thread.java: 829)
Caused by: com.vmware. vcf.pasawordmanager.exception.PasswordUpdateException: Unable to get NSX-T Transport Nodes from NSX-T Cluster
at com.vmware.vcf.passwordmanager.helper.NsxtApiUtil.getNsxtTransportNodeByFqdn (NaxtApiUtil. java:343)

     

Cause

Mismatched hostname of Edge Node in SDDC Manager Platform DB

Resolution

  1. SSH to sddc manager with vcf user and su to root
  2. Run the below command on SDDC Manager to fetch all the edge nodes deployed 
    curl -X GET -k -s -w "\n" -u 'admin:<password>' https://<nsxt-manager-fqdn>/api/v1/transport-nodes | json_pp


    Find the id of the edge node from the output

  3. Run the below command to list the details of the edge node for which password remediate is failing 
    curl -X GET -k -s -w "\n" -u 'admin:<password>' https://<nsxt-manager-fqdn>/api/v1/transport-nodes/<id of edge node from step# 2> | json_pp

    The hostname here should exactly match the hostname on SDDC DB & in the NSXT Edge CLI ---> Match FQDN / Short name / Lower or upper case.

  4. Update the edge node name in SDDC Manager Platform database
    1. Take snapshot of SDDC Manager VM
    2. Connect to platform database 
      psql -h localhost -U postgres -d platform

       

    3. Expand the display (optional) 
      \x

       

    4. List the edge node details 
      select id,status,name,nsxt_edge_nodes from nsxt_edge_cluster;


      Sample output

      id 					| a8c5####-####-####-####-########1aa7
status 				| ACTIVE
name 				| edge-cluster.example.com
nsxt_edge_nodes		| [{"vmManagement IpAddress": "192.0.2.2", "vmHostname": "edge01.example.com", "sourceId": "3d72####-####-####-####-########0c4b", "id": "5f71####-####-####-####-########082c"}, {"vmManagement IpAddress": "192.0.2.3", "vmHostname": "edge02.example.com", "sourceId": "935e####-####-####-####-########d9a2", "id": "c10a####-####-####-####-########ec6b"}]

       

    5. Update edge node names (updating edge node names in "vmHostname") 
      update nsxt_edge_cluster set nsxt_edge_nodes='<edge node names from Step # 3>' where id='<id of edge cluster from step # 4(d)>';


      Sample

      update nsxt_edge_cluster set nsxt_edge_nodes='[{"vmManagement IpAddress": "192.0.2.2", "vmHostname": "EDGE01.example.com", "sourceId": "3d72####-####-####-####-########0c4b", "id": "5f71####-####-####-####-########082c"}, {"vmManagement IpAddress": "192.0.2.3", "vmHostname": "edge02.example.com", "sourceId": "935e####-####-####-####-########d9a2", "id": "c10a####-####-####-####-########ec6b"}]' where id='a8c5####-####-####-####-########1aa7';

       

  5. Refresh the SDDC Manager UI, retry password remediation for NSX-T Edge nodes.
  6. Once successful please reset the db entries back to the original values as it was. This is to make sure none of the other workflows would break. 
Powered by Wolken Software