Failing to update/remediate/rotate NSX-T edge node root password with error: Failed to find a VM having prefix match of DNS name with FQDN <edge_node_FQDN>

Products

• VMware Cloud Foundation

Issue/Introduction

Password management operations fail for the NSX-T edge node root user. 

Symptoms:

1. Below operations for password management  from the SDDC UI for NSXT-T edge node root user fails with the UI error: Unable to obtain SSH connectivity to entity: <edge node fqdn>"
   • Update password 
   • Remediate either the new password after changing on the edge node or already saved passsword in the SDDC DB
   • Rotate passsword
2. The state is 'Disconnected' for the root user in SDDC Manager Password Management
3. All the password management operations works fine for admin and audit user. 

 

• Below are the entries in the operationsmanager.log

  yyyy-mm-ddThh:mm:ss WARN [vcf_om,59a4058823f98c10,f3c5] [c.v.v.p.helper.GuestProgramService,om-exec-30] Failed to find VM by complete FQDN, checking if VM DNS name prefixes the FQDN
  yyyy-mm-ddThh:mm:ss DEBUG [vcf_om,59a4058823f98c10,f3c5] [c.v.v.p.helper.GuestProgramService,om-exec-30] Fetched list of powered on virtual machines [] whose DNS name prefixes FQDN <nsx_edge_fqdn> from cluster <cluster_name>
  yyyy-mm-ddThh:mm:ss ERROR [vcf_om,59a4058823f98c10,f3c5] [c.v.v.p.helper.GuestProgramService,om-exec-30] Failed to find a VM having prefix match of DNS name with FQDN <nsx_edge_fqdn>
  yyyy-mm-ddThh:mm:ss ERROR [vcf_om,59a4058823f98c10,f3c5] [c.v.v.p.helper.GuestProgramService,om-exec-30] Exception occurred while finding vmMor for <nsx_edge_fqdn> : Failed to find powered-on VM with FQDN <nsx_edge_fqdn>
  yyyy-mm-ddThh:mm:ss DEBUG [vcf_om,59a4058823f98c10,f3c5] [c.v.v.p.helper.GuestProgramService,om-exec-30] Failed to run guest program for <nsx_edge_fqdn>
  yyyy-mm-ddThh:mm:ss DEBUG [vcf_om,59a4058823f98c10,f3c5] [c.v.e.s.c.c.v.vsphere.VsphereClient,om-exec-30] Destroying 1 open views
  yyyy-mm-ddThh:mm:ss WARN [vcf_om,59a4058823f98c10,f3c5] [c.v.v.v.c.h.i.HttpConfigurationCompilerBase$ConnectionMonitorThreadBase,om-exec-30] Shutting down the connection monitor.
  yyyy-mm-ddThh:mm:ss WARN [vcf_om,0000000000000000,0000] [c.v.v.v.c.h.i.HttpConfigurationCompilerBase$ConnectionMonitorThreadBase,VLSI-client-connection-monitor-1047] Interrupted, no more connection pool cleanups will be performed.
  yyyy-mm-ddThh:mm:ss ERROR [vcf_om,59a4058823f98c10,f3c5] [c.v.v.p.u.changers.NsxtEdgeChanger,om-exec-30] Exception in doTest of com.vmware.vcf.passwordmanager.update.changers.NsxtEdgeChanger
  com.vmware.vcf.passwordmanager.exception.Fault: Failed executing command in VM <nsx_edge_fqdn>
   at com.vmware.vcf.passwordmanager.update.changers.NsxtEdgeChanger.doInVmwareToolsForNsxtEdge(NsxtEdgeChanger.java:580)
   at com.vmware.vcf.passwordmanager.update.changers.NsxtEdgeChanger.runCommandInNsxtEdge(NsxtEdgeChanger.java:533)
   at com.vmware.vcf.passwordmanager.update.changers.NsxtEdgeChanger.doTest(NsxtEdgeChanger.java:99)
   at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.updateAsync(AbstractPasswordChanger.java:430)
  
  yyyy-mm-ddThh:mm:ss ERROR [vcf_om,59a4058823f98c10,f3c5] [c.v.v.p.u.changers.NsxtEdgeChanger,om-exec-30] Exception occurred while testing NSXT Edge node credentials
  com.vmware.vcf.passwordmanager.exception.PasswordUpdateException: Failed executing command in VM <nsx_edge_fqdn>
   at com.vmware.vcf.passwordmanager.update.changers.NsxtEdgeChanger.runCommandInNsxtEdge(NsxtEdgeChanger.java:547)
   at com.vmware.vcf.passwordmanager.update.changers.NsxtEdgeChanger.doTest(NsxtEdgeChanger.java:99)
   
  Caused by: com.vmware.vcf.passwordmanager.exception.PasswordUpdateException: Failed executing command in VM <nsx_edge_fqdn>
   at com.vmware.vcf.passwordmanager.update.changers.NsxtEdgeChanger.runCommandInNsxtEdge(NsxtEdgeChanger.java:547)
   at com.vmware.vcf.passwordmanager.update.changers.NsxtEdgeChanger.doTest(NsxtEdgeChanger.java:99)
   ... 9 common frames omitted
  yyyy-mm-ddThh:mm:ss DEBUG [vcf_om,59a4058823f98c10,f3c5] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-30] Error Message : Failed executing command in VM <nsx_edge_fqdn>, Error Token : 5BBTQJ, Error Cause : {}
  yyyy-mm-ddThh:mm:ss DEBUG [vcf_om,59a4058823f98c10,f3c5] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-30] About to mark resource state as error...
  yyyy-mm-ddThh:mm:ss DEBUG [vcf_om,59a4058823f98c10,f3c5] [c.v.v.p.r.AbstractPasswordTransactionExecutor,om-exec-30] Password operation failed for root

Environment

VMware Cloud Foundation 4.5.x

Cause

Below are the possible scenarios which could this issue:

Scenario 1:
This issue is because of case sensitive characters. The DNS name of the edge node VM in VC is not in the same case when compared with the vmHostname in nsxt_edge_cluster table in the SDDC DB.

Scenario 2:
The Edge VM is not present in the Cluster as identified in Workload Domain > Edge Cluster > vSphere hosting Cluster

Resolution

Resolution for Scenario 1:

• Login in to VC, and check the DNS name for the edge VM. If the case(lower or upper) is not similar to the vmHostname in the nsxt_edge_cluster table in the SDDC DB, then we need to update hostname by following the below steps:
  1. Take snapshot of Edge node VM
  2. Take SSH to Edge node VM and login with root.
  3. Run: hostname <dns name which we see in VC in case similar to vmHostname in SDDC DB>
     

     Example:
     vmHostname = nsxt_edge_node
     VC DNS = NSXT_EDGE_NODE
     
     Command: hostname nsxt_edge_node

     
     
  4. Wait for 1-2 mins and refresh VC UI, we should now see DNS name to be in respective case.
  5. Retry the password management operation from the SDDC. 

Scenario 2:

• Login to SDDC Manager UI
• Browse to Workload Domain > [Domain-name] > Edge Cluster
• Check the vSphere Hosting Cluster name  and verify if the Edge VM is actually in those vSphere clusters in vCenter Inventory.
• If Edge VM is a different vSphere Cluster and not matching with vSphere Hosting Cluster name in SDDC then vMotion the VM (If Edge VM has no network/vlan configuration issues) to the same cluster which is showing in SDDC UI's vSphere Hosting Cluster.

*** If the resolution steps for Scenario 2 cannot be implemented because of Environmental limitations then contact Broadcom Support for assistance

Additional Information

Impact/Risks:
The NSX-T edge node root user is in "Disconnected" state and all the password management operations for Edge node root user fail from the SDDC.