NSX Admin, Root and Audit accounts are in a disconnected state, remediation shows as successful but rotation fails

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

Password rotation and remediation tasks are not running normally and NSX Edge node accounts remain in disconnected state.
The password remediate function continuously returns an error message in the SDDC UI when attempted. “Password management operation failed.
Confirmed no issues with SSH Host keys for NSX Edge nodes per KB SSH connection fails with "reject HostKey" or "Primary psc init failed" errors.
When password rotation/remediation for the admin, root, or audit accounts fails, entries similar to the following appear in /var/log/vmware/vcf/operationsmanager/operationsmanager.log:

####-##-##T##:##:##:###+0000 ERROR [vcf_om,#############################] [c.v.v.p.u.changers.NsxtEdgeChanger,om-exec-24] Exception occurred while retrieving password expiry information for NSXT Edge
com.vmware.vcf.passwordmanager.exception.PasswordUpdateException: Unable to get NSX Transport Nodes from NSX Cluster EXAMPLE_CLUSTER.
at com.vmware.vcf.passwordmanager.helper.NsxtApiUtil.getNsxtTransportNodeByFqdn(NsxtApiUtil.java:572)
at com.vmware.vcf.passwordmanager.update.changers.NsxtEdgeChanger.getPasswordExpiry(NsxtEdgeChanger.java:269)

and/or

####-##-##T##:##:##:###+0000 ERROR [vcf_om,#############################] [c.v.v.p.s.PasswordExpirationService,om-exec-2] Expiry retrieval status : UNKNOWN , Diagnostic message : {"errorCode":"PASSWORD_MANAGER_RETRIEVE_PASSWORD_EXPIRY_FAILED","arguments":["VM-NSX_EDGE-NODE1.vmwareexample.cloud"],"errorMessage":"Unable to get NSX Transport Nodes from NSX Cluster EXAMPLE_CLUSTER.","referenceToken":"#####","remediationMessage":"Please verify if the account credentials can be used to login to the resource. You might need to fix the workflow(s) for resources marked in error state. If the password of the account has expired, manually reset the password in the product and then perform a REMEDIATE operation in the SDDC Manager, to update its stored copy of the password."}

The following command reveals that the hostname value for the affected node (NODE1 in this example) is not the full FQDN:

# curl -X GET -k -s -w "\n" -u 'admin:<password>' https://<nsxt-manager-fqdn>/api/v1/transport-nodes | json_pp

Example 1:

For NODE2: WORKING example

}
},
"deployment_type" : "VIRTUAL_MACHINE",
"display_name" : "VM-NSX_EDGE-NODE2.vmwareexample.cloud",
"external_id" : "#########################",
"id" : "##########################",
"ip_addresses" : [
"1.1.1.1"
],
"node_settings" : {
"allow_ssh_root_login" : true,
"dns_servers" : [
"1.1.1.1",
"1.1.1.1"
],
"enable_ssh" : true,
"enable_upt_mode" : false,
"hostname" : "VM-NSX_EDGE-NODE2.vmwareexample.cloud",
"ntp_servers" : [
"1.1.1.1",
"1.1.1.1"
],
"search_domains" : [
"vmwareexample.cloud"

For NODE 1: NOT WORKING

}
},
"deployment_type" : "VIRTUAL_MACHINE",
"display_name" : "VM-NSX_EDGE-NODE1", <<<<<<<<<<<Doesn’t match SDDC
"external_id" : "#########################",
"id" : "##########################",
"ip_addresses" : [
"1.1.1.1"
],
"node_settings" : {
"allow_ssh_root_login" : true,
"dns_servers" : [
"1.1.1.1",
"1.1.1.1"
],
"enable_ssh" : true,
"enable_upt_mode" : false,
"hostname" : "VM-NSX_EDGE-NODE1", <<<<<<<<<<<Doesn’t match SDDC

"ntp_servers" : [
"1.1.1.1",
"1.1.1.1"
],
"search_domains" : [
"vmwareexample.cloud"

Method 2:

ssh into the nsx manager console as admin level user.

Run the get nodes command.

get nodes

UUID Type IP Address IPv6 Address Hostname/FQDN Display Name

<#########UUID#####################> edg <IP_address> not configured lab-edge-node-01a lab-edge-node-01a <<<<<<<<<<<Doesn’t match SDDC

<#########UUID#####################> edg <IP_address> not configured lab-edgenode-02a.lab-domain.local lab-edge-node-02a.lab-domain.local

Environment

VMware Cloud Foundation 5.2.x
VMware Cloud Foundation 9.x

Cause

The Hostname and/or Display Name value specified for the NSX node is incorrect, resulting in a discrepancy with the records contained within the SDDC database. Consequently, the API calls required to execute password rotations and remediations fail, as they are unable to establish a proper connection to retrieve the accurate expiration dates.

Resolution

Updating the NSX edge node hostname and or display name to match what's in SDDC in the NSX UI, resolves this issue and allows password rotation and remediation tasks to proceed in SDDC.

Follow the steps below: