NSX-T ESXi host shows as not installed "Host is not reachable. Cannot complete login due to an incorrect username or password"
search cancel

NSX-T ESXi host shows as not installed "Host is not reachable. Cannot complete login due to an incorrect username or password"

book

Article ID: 322555

calendar_today

Updated On:

Products

VMware NSX Networking

Issue/Introduction

Symptoms:
  • In 'System - Fabric - Nodes - Host Transport Nodes' the host install fails with below error:
"Host is not reachable. Cannot complete login due to an incorrect username or password".
  • After you log into the NSX-T manager UI, the host may toggle between NSX Install successful / NSX Install failed.
  • All hosts present in the cluster shows installation was successful, but the cluster shows Preparation Failed.
  • Log message: /var/log/proton/nsxapi.log
2023-01-31T02:29:54.826Z  INFO http-nio-127.0.0.1-7440-exec-30 InstallFailed 138228 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" reqId="e95947f2-8464-4189-a89b-416b9fd29df1" subcomp="manager" username="admin"] Issue INSTALL_FAILED observed for node : 32e6a6ee-b3fb-4227-b3cf-51ec7739ea35. Msg : Failed to install software on host. Host 192.168.1.1 not reachable. java.rmi.RemoteException: VI SDK invoke exception:SoapFaultException [Cannot complete login due to an incorrect user name or password.] : java.rmi.RemoteException: VI SDK invoke exception:SoapFaultException [Cannot complete login due to an incorrect user name or password.]
2023-01-31T02:45:58.541Z  INFO pool-856-thread-4 InstallFailed 138228 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Issue INSTALL_FAILED observed for node : 32e6a6ee-b3fb-4227-b3cf-51ec7739ea35. Msg : Failed to install software on host. Host 192.168.1.1 not reachable. java.rmi.RemoteException: VI SDK invoke exception:SoapFaultException [Cannot complete login due to an incorrect user name or password.] : java.rmi.RemoteException: VI SDK invoke exception:SoapFaultException [Cannot complete login due to an incorrect user name or password.]
  • API call to review the TN (Transport Node) status shows the status UNKNOWN and install failed GET /api/v1/transport-nodes/<TN-UUID>/status
    • For example:
curl -k -u admin:password -X GET https://192.168.1.1/api/v1/transport-nodes/32e6a6ee-b3fb-4227-b3cf-51ec7739ea35/status
{
  "node_uuid" : "32e6a6ee-b3fb-4227-b3cf-51ec7739ea35",
  "node_display_name" : "192.168.1.1",
  "status" : "UNKNOWN",
  "mgmt_connection_status" : "UNKNOWN",
  "node_status" : {
    "mpa_connectivity_status" : "UNKNOWN",
    "mpa_connectivity_status_details" : "Client has not responded to heartbeats yet",
    "lcp_connectivity_status" : "UNKNOWN",
    "lcp_connectivity_status_details" : [ ],
    "host_node_deployment_status" : "INSTALL_FAILED",
    "inventory_sync_paused" : false,
    "software_version" : ""
  }
  • API call to review host state shows the error presented GET /api/v1/transport-nodes/<TN-UUID>/state
    • For example:
curl -k -u admin:password -X GET https://192.168.1.1/api/v1/transport-nodes/32e6a6ee-b3fb-4227-b3cf-51ec7739ea35/state
{
  "transport_node_id" : "32e6a6ee-b3fb-4227-b3cf-51ec7739ea35",
  "maintenance_mode_state" : "DISABLED",
  "node_deployment_state" : {
    "state" : "failed",
    "details" : [ {
      "sub_system_id" : "32e6a6ee-b3fb-4227-b3cf-51ec7739ea35",
      "state" : "failed",
      "failure_message" : "Failed to install software on host. Host 1.1.1.1 not reachable. java.rmi.RemoteException: VI SDK invoke exception:SoapFaultException [Cannot complete login due to an incorrect user name or password.] : java.rmi.RemoteException: VI SDK invoke exception:SoapFaultException [Cannot complete login due to an incorrect user name or password.]",
      "failure_code" : 26080
    } ]
  },
  "deployment_progress_state" : {
    "progress" : 0,
    "current_step_title" : "Preparing Installation"
  },
  "state" : "pending"


Environment

VMware NSX-T
VMware NSX-T Data Center

Cause

This issue is due to a race condition, which can occur under heavy load.

Resolution

This is a known issue impacting NSX-T.

Workaround:
  • Find the transport node UUID for the failed host, using the below API call:
  • For On prem:
GET /policy/api/v1/infra/sites/default/enforcement-points/default/host-transport-nodes
  • For VMC:

GET /policy/api/v1/infra/sites/default/enforcement-points/vmc-enforcementpoint/host-transport-nodes

  • This will return a list of all transport nodes:
{
    "results": [
        {
            "node_deployment_info": {
                "fqdn": "esx-04.corp.local",
                "ip_addresses": [
                    "192.168.115.34"
... some text omitted for clarity.
            "resource_type": "HostTransportNode",
            "id": "56e064ec-45ea-4b29-8394-8f4e735066b3",
            "display_name": "esx-04.corp.local",
  • From the result we are looking for the node UUID:
"id": "56e064ec-45ea-4b29-8394-8f4e735066b3"
  • If the TN has 2 UUID's, note those. You can use the TN management IP address to confirm it is the same TN:
"ip_addresses": [
"192.168.115.34"
  • From the NSX-T UI 'System - Fabric - Nodes - Host Transport Nodes' remove NSX-T from the failed host, select the host and click 'REMOVE NSX'.
  • If a TNP (Transport Node Profile) is used, in vCenter move the host from the NSX-T prepared cluster to an unprepared cluster.
  • Then using the UUID's discovered for the failed node to force delete the node with the following API call:
DELETE https://<mgr-ip>/api/v1/transport-nodes/<tn_id_1>?unprepare_host=false&force=true
  • Repeat the above force delete API call for any secondary UUID's found for the same TN.
  • Then either prepare the TN again, if not using a TNP or if using a TNP, move the transport node back into the cluster to get prepared again.


Powered by Wolken Software