NSX Manager Root Password Remediation on SDDC Manager fails with error "String.split(String, int)" because "fqdn" is null"

Products

• VMware SDDC Manager

Issue/Introduction

• NSX Manager root account is disconnected. 
  
  
• Upon remediating/updating the root password on the SDDC, it fails with the  error - Cannot invoke "String.split(String, int)" because "fqdn" is null
  

operationsmanager.log - /var/log/vmware/vcf/operationsmanager/operationsmanager.log 

YYYY-MM-DDT03:34:37.948+0000 DEBUG [vcf_om,682bf84d5400954945fad8525e5767fb,a404] [c.v.e.s.c.c.v.vsphere.VcManagerBase,om-exec-28] Finding vms with host name nsx_manager_fqdn
YYYY-MM-DDT03:34:37.953+0000 WARN  [vcf_om,682bf84d5400954945fad8525e5767fb,a404] [c.v.v.p.helper.GuestProgramService,om-exec-28] Could not find VMs with hostname: nsx_manager_fqdn. (vmodl.fault.SystemError) {
   faultCause = null,
   faultMessage = null,
   reason = Missing non-optional return value
}
YYYY-MM-DDT03:34:37.953+0000 DEBUG [vcf_om,682bf84d5400954945fad8525e5767fb,a404] [c.v.v.p.helper.GuestProgramService,om-exec-28] 0 powered on VMs found with FQDN: nsx_manager_fqdn
YYYY-MM-DDT03:34:37.982+0000 DEBUG [vcf_om,682bf84d5400954945fad8525e5767fb,a404] [c.v.e.s.c.c.v.vsphere.VsphereClient,om-exec-28] Destroying 1 open views
YYYY-MM-DDT03:34:37.992+0000 ERROR [vcf_om,682bf84d5400954945fad8525e5767fb,a404] [c.v.v.p.u.c.NsxtManagerSshChanger,om-exec-28] The credential test failed for root@nsx_manager_fqdn due to java.lang.NullPointerException: Cannot invoke "String.split(String, int)" beca
use "fqdn" is null
YYYY-MM-DDT03:34:37.992+0000 ERROR [vcf_om,682bf84d5400954945fad8525e5767fb,a404] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-28] Cannot invoke "String.split(String, int)" because "fqdn" is null

• NSX Manager is deployed on the Workload domain instead of the Management domain. 

Environment

SDDC Manager 5.2.1

Cause

• SDDC Manager tries to find the NSX Manager on the Management Domain instead of the Workload Domain.
• The logic in the code for locating VMs is that it first searches for the VMs within the  Management vCenter, and if the VMs are not found in the  Management vCenter, then the search is extended to include any vCenter servers that are connected to the NSX Manager instance.
• If there is any VM in the Management vCenter that has an FQDN with a null value, this issue is encountered.
• DNS Name (hostname/FQDN) had never been populated/cached for the VM, and thus, it returns a null value.

Resolution

The issue is a known issue and has been fixed in VCF 9.0. 

Workaround :

• Power Off and Power ON the VM. This will populate the DNS name for the VM. 

or 

• Delete the VM from the disk OR Remove the VM from inventory if it is not in Use.

Additional Information

Note: To check which VM in the VC has null as FQDN run the below command on the management VC:

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
SELECT file_name FROM vpx_vm WHERE dns_name IS NULL;