NSX Admin, Root and Audit accounts are in a disconnected state, remediation shows as successful but rotation fails

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• Password rotation and remediation tasks are not running normally and NSX Edge node accounts remain in disconnected state. 
• Confirmed no issues with SSH Host keys for NSX Edge nodes per KB How to update the SSH host keys on the SDDC Manager.
• When attempting to rotate passwords for the admin, root or audit accounts, you may see entries similar to below in /var/log/vmware/vcf/operationsmanager/operationsmanager.log:

####-##-##T##:##:##:###+0000 ERROR [vcf_om,#############################] [c.v.v.p.u.changers.NsxtEdgeChanger,om-exec-24] Exception occurred while retrieving password expiry information for NSXT Edge
com.vmware.vcf.passwordmanager.exception.PasswordUpdateException: Unable to get NSX Transport Nodes from NSX Cluster EXAMPLE_CLUSTER.
        at com.vmware.vcf.passwordmanager.helper.NsxtApiUtil.getNsxtTransportNodeByFqdn(NsxtApiUtil.java:572)
        at com.vmware.vcf.passwordmanager.update.changers.NsxtEdgeChanger.getPasswordExpiry(NsxtEdgeChanger.java:269)

and/or 

####-##-##T##:##:##:###+0000 ERROR [vcf_om,#############################] [c.v.v.p.s.PasswordExpirationService,om-exec-2] Expiry retrieval status : UNKNOWN ,  Diagnostic message : {"errorCode":"PASSWORD_MANAGER_RETRIEVE_PASSWORD_EXPIRY_FAILED","arguments":["VM-NSX_EDGE-NODE1.vmwareexample.cloud"],"errorMessage":"Unable to get NSX Transport Nodes from NSX Cluster EXAMPLE_CLUSTER.","referenceToken":"#####","remediationMessage":"Please verify if the account credentials can be used to login to the resource. You might need to fix the workflow(s) for resources marked in error state. If the password of the account has expired, manually reset the password in the product and then perform a REMEDIATE operation in the SDDC Manager, to update its stored copy of the password."}

• When running below command, you may find that the affected node (NODE1 in our example) hostname value is not the full FQDN: 

# curl -X GET -k -s -w "\n" -u 'admin:<password>' https://<nsxt-manager-fqdn>/api/v1/transport-nodes | json_pp 

Example: 

For NODE2: WORKING example

       }
            },
            "deployment_type" : "VIRTUAL_MACHINE",
            "display_name" : "VM-NSX_EDGE-NODE2",
            "external_id" : "#########################",
            "id" : "##########################",
            "ip_addresses" : [
               "1.1.1.1"
            ],
            "node_settings" : {
               "allow_ssh_root_login" : true,
               "dns_servers" : [
                  "1.1.1.1",
                  "1.1.1.1"
               ],
               "enable_ssh" : true,
               "enable_upt_mode" : false,
               "hostname" : "VM-NSX_EDGE-NODE2.vmwareexample.cloud",
               "ntp_servers" : [
                  "1.1.1.1",
                  "1.1.1.1"
               ],
               "search_domains" : [
                  "vmwareexample.cloud"

For NODE 1: NOT WORKING

  }
            },
            "deployment_type" : "VIRTUAL_MACHINE",
            "display_name" : "VM-NSX_EDGE-NODE1",
            "external_id" : "#########################",
            "id" : "##########################",
            "ip_addresses" : [
               "1.1.1.1"
            ],
            "node_settings" : {
               "allow_ssh_root_login" : true,
               "dns_servers" : [
                  "1.1.1.1",
                  "1.1.1.1"
               ],
               "enable_ssh" : true,
               "enable_upt_mode" : false,
               "hostname" : "VM-NSX_EDGE-NODE1",
               "ntp_servers" : [
                  "1.1.1.1",
                  "1.1.1.1"
               ],
               "search_domains" : [
                  "vmwareexample.cloud"

Environment

• VMware Cloud Foundation 5.2.x
• VMware Cloud Foundation 9.x 

Cause

The Hostname value for the NSX node is incorrect and may show as the display name instead of the actual fully-qualified domain name (FQDN). Due to this, the API calls required to rotate the passwords fail as they cannot properly connect to retrieve correct expiration dates.

Resolution

To solve this issue, you should update the NSX edge nodes hostname value to be the complete FQDN instead of the display- or shortname in the NSX UI and reattempt the password rotate/remediate tasks from SDDC.

Follow the steps below:

1. Navigate to NSX UI > System > Fabric > Nodes.
2. Select Edge and from the ACTIONS menu choose "Change node settings"
3. Change the FQDN at the top and save the change.
4. Open the ACTIONS menu again and select "Sync Edge Node Configuration".

5. Try to remediate the passwords/rotate them from SDDC UI.

 

Additional Information

SDDC Manager password remediation for NSX-T Edge nodes failing with Error: Unable to get transport nodes from NSX-T cluster