Health check operations fails in SDDC for NSX edge nodes - "Could not find VMs with FQDN" error in operationsmanager.log

Products

• VMware SDDC Manager

Issue/Introduction

SDDC Manager Health-Check operation fails for Password-check.

Running /opt/vmware/sddc-support/sos --password-health from the SSH session reports "Failed to get details" for the NSX edge nodes 

Password Expiry Status : RED
+-----+-------------------------------+--------+-------------------+--------------+-----------------+-----------------------+
| SL# |           Component           |  User  | Last Changed Date | Expiry Date  | Expires in Days |         State         |
+-----+-------------------------------+--------+-------------------+--------------+-----------------+-----------------------+
|  5  |   NSX : nsx_edge01_FQDN       |  root  |         -         |      -       |        -        | Failed to get details |
|  6  |   NSX : nsx_edge02_FQDN        |  root  |         -         |      -       |        -        | Failed to get details |

/var/log/vmware/vcf/operationsmanager/operationsmanager.log have the below loggings:

[c.v.e.s.c.c.v.vsphere.VcManagerBase,om-exec-13] Finding vms with host name nsx_edge01_FQDN
yyyy-mm-ddT01:01:01.506+0000 DEBUG [vcf_om,67f7184dc2da6fe9bb68a37d96219614,1820] [c.v.v.p.u.changers.NsxtEdgeChanger,om-exec-7] vm found: nsx_edge01
yyyy-mm-ddT01:01:01.506+0000 DEBUG [vcf_om,67f7184d447aa6b9b012e67125621147,6246] [c.v.v.p.helper.GuestProgramService,om-exec-17] 1 VMs found with hostname: nsx_edge01
yyyy-mm-ddT01:01:01.509+0000 WARN  [vcf_om,67f7184df3ccdd3ce0e3dc5bb595e578,6c55] [c.v.v.p.helper.GuestProgramService,om-exec-5] 
Could not find VMs with FQDN: nsx_edge01_FQDN
(vmodl.fault.SystemError) {
   faultCause = null,
   faultMessage = null,
   reason = Missing non-optional return value
}

[c.v.e.s.c.c.v.vsphere.VcManagerBase,om-exec-13] Finding vms with host name nsx_edge02_FQDN
yyyy-mm-ddT01:01:01.512+0000 DEBUG [vcf_om,67f7184d669a27afcdac94d3f2f33f38,94b4] [c.v.v.p.u.changers.NsxtEdgeChanger,om-exec-14] vm found: nsx_edge02
yyyy-mm-ddT01:01:01.506+0000 DEBUG [vcf_om,67f7184dc2da6fe9bb68a37d96219614,1820][c.v.v.p.helper.GuestProgramService,om-exec-15] 1 VMs found with hostname: nsx_edge02
yyyy-mm-ddT01:01:01.514+0000 WARN  [vcf_om,67f7184d11b39b1c454ad01f836f989d,8ea7] [c.v.v.p.helper.GuestProgramService,om-exec-13] 
Could not find VMs with FQDN: nsx_edge02_FQDN
(vmodl.fault.SystemError) {
   faultCause = null,
   faultMessage = null,
   reason = Missing non-optional return value
}

Environment

VMware Cloud Foundation 5.x

Cause

The DNS name for the NSX Edge node VM in vCenter does not match with the vmHostname in nsxt_edge_cluster table in the SDDC DB.

Resolution

1. Take SSH to Edge node VM and login with root.
   
   
2. Update the DNS to match the vmHostname in SDDC DB
   Command to check vmHostname from SDDC Manager DB: psql -h localhost -U postgres -d platform -c "select * from nsxt_edge_cluster;"
   

   hostname nsxt_edge_node

3. Refresh the vCenter UI to ensure the DNS name is reflecting to the updated FQDN.

Additional Information

If there is case sensitivity mismatch for the hostname, refer KB  Failing to update/remediate/rotate NSX-T edge node root password with error: Failed to find a VM having prefix match of DNS name with FQDN <edge_node_FQDN>