Unable to add ESXi host to vCenter Server with the following error "Cannot complete login due to incorrect user name or password"

Products

VMware vSphere ESXi

Issue/Introduction

When attempting to add the ESXi host to vCenter Server, it fails with the following error message :

Within the /var/log/vmware/vpxd/vpxd.log file on vCenter Server, following log entries are observed :

YYYY-MM-DDTHH:MM:SS error vpxd[06335] [Originator@6876 sub=TrustedInfrastructure.HostConfig opID=rd-#######-#####-#####-#####-ef] [CreateApiProvider::errorCb] Providers stack failed: Error:
-->    system_error
--> Messages:
-->    vapi.invalid.result.code<Recv of frame failed with code: 503 Service Unavailable>
-->
YYYY-MM-DDTHH:MM:SS error vpxd[05779] [Originator@6876 sub=TrustedInfrastructure.HostConfig opID=rd-#######-#####-#####-#####-ef] [StsUploader] Failed to set STS certificates to host '<IP address of ESXi host>'. Error:
--> Error:
-->    system_error
--> Messages:
-->    vapi.invalid.result.code<Recv of frame failed with code: 503 Service Unavailable>
-->
YYYY-MM-DDTHH:MM:SS info vpxd[05779] [Originator@6876 sub=HostCnx opID=rd-#######-#####-#####-#####-ef] [VpxdHostCnx::RemoveConnection] cnx: 52#####-####-####-####-########ec
YYYY-MM-DDTHH:MM:SSerror vpxd[05779] [Originator@6876 sub=InvtHost opID=rd-#######-#####-#####-#####-ef] Failed to reconnect to cleanup before host removal: N3Vim5Fault12InvalidLogin9ExceptionE(Fault cause: vim.fault.InvalidLogin
--> )

YYYY-MM-DDTHH:MM:SS error vpxd[1985155] [Originator@6876 sub=Default opID=rd-#######-#####-#####-#####-ef] [VpxLRO] -- ERROR task-<task_ID>) -- group-h##### -- vim.Folder.addStandaloneHost: :vim.fault.InvalidLogin
--> Result:
--> (vim.fault.InvalidLogin) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = ""
--> }
--> Args:
-->
--> Arg spec:
--> (vim.host.ConnectSpec) {
-->    hostName = "<IP Address of ESXi host>",
-->    port = <unset>,
-->    sslThumbprint = "<Certificate Thumbprint of ESXi host>",
-->    sslCertificate = <unset>,
-->    userName = "root",
-->    password = (not shown),
-->    vmFolder = 'vim.Folder:43#####-####-####-####-#########89:group-v#####',
-->    force = true,
-->    vimAccountName = "vpxuser",
-->    vimAccountPassword = (not shown),
-->    managementIp = <unset>,
-->    lockdownMode = "lockdownDisabled",
-->    hostGateway = (vim.host.GatewaySpec) null
--> }
--> Arg compResSpec:
-->
--> Arg addConnected:
--> true

YYYY-MM-DDTHH:MM:SS  info vpxd[1984728] [Originator@6876 sub=vpxLro opID=wcp-licenseRefreshMonitor-b4] [VpxLRO] -- FINISH lro-19974
YYYY-MM-DDTHH:MM:SS warning vpxd[1985227] [Originator@6876 sub=TrustedInfrastructure.HostConfig opID=rd-#######-#####-#####-#####-ef] [CreateOrUpdate] Policy with local user name vpxuser already exists on host <IP Address of ESXi host>. Exception: N4Vpxd7Clients22AlreadyExistsExceptionE(Error:
-->    com.vmware.vapi.std.errors.already_exists
--> No messages!
--> )

YYYY-MM-DDTHH:MM:SS info vpxd[1985227] [Originator@6876 sub=vmomi.soapStub[114] opID=rd-#######-#####-#####-#####-ef] SOAP request returned HTTP failure; <SSL(<io_obj p:0x00007fd6e403e000, h:121, <TCP '<IP address of vCenter Server> : 51416'>, <TC
P '<IP Address of ESXi host> : 443'>>), /sdk>, method: createUser; code: 500(Internal Server Error); fault: (vim.fault.AlreadyExists) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    name = "vpxuser"
-->    msg = "Received SOAP response fault from [<SSL(<io_obj p:0x00007fd6e403e000, h:121, <TCP '<IP address of vCenter Server> : 51416'>, <TCP '<IP Address of ESXi host> : 443'>>), /sdk>]: createUser
--> The specified key, name, or identifier 'vpxuser' already exists."
--> }

When performing a vmkping test from the ESXi host to the vCenter Server, the host is unable to transmit packets beyond a specific MTU threshold, which was lower than the configured MTU size on both the ESXi host and vCenter Server.:

In the example below, the ESXi host could successfully send packets up to an MTU size of 1372 bytes, but any packets exceeding this size consistently failed.

Successful test: 

vmkping -I vmk0 <vCenter Server IP> -d -s 1372
PING <vCenter Server IP> 1372 data bytes
1380 bytes from <vCenter Server IP>: icmp_seq=0 ttl=63 time=21.351 ms
1380 bytes from <vCenter Server IP>: icmp_seq=1 ttl=63 time=20.274 ms
1380 bytes from <vCenter Server IP>: icmp_seq=2 ttl=63 time=20.533 ms

--- <vCenter Server IP> ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss

Failed test: 

vmkping -I vmk0 <vCenter Server IP> -d -s 1373
PING <vCenter_IP> 1373 data bytes

--- <vCenter Server IP> ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

Environment

VMware vCenter Server

VMware vSphere ESX

Cause

The issue happens because of MTU fragmentation or mismatch between the ESXi host and the vCenter Server during the secure communication handshake.

Resolution

To resolve the issue, verify and correct the MTU configuration across all layers of the management network path between the vCenter Server and ESXi host.
If the MTU is not set correctly at ESXi level, change it to appropriate value on both the management vmkernel adapter and vSwitch :
- Change the Size of the MTU on a vSphere Standard Switch
- Edit a VMkernel Adapter Configuration
Verify that the MTU is correctly set on the physical networking layer and ensure the path supports 1500 MTU or higher (if using Jumbo frames).

After correcting the MTU values, verify the connection again from ESXi host to vCenter Server with the following command -

vmkping -I vmk0 <vCenter Server IP> -d -s 1472

Expected Output

PING <vCenter Server IP> 1472 data bytes
1472 bytes from <vCenter Server IP>: icmp_seq=0 ttl=63 time=21.351 ms
1472 bytes from <vCenter Server IP>: icmp_seq=1 ttl=63 time=20.274 ms
1472 bytes from <vCenter Server IP>: icmp_seq=2 ttl=63 time=20.533 ms

--- <vCenter Server IP> ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss

Add the ESXi host again to the vCenter Server.

Additional Information

Unable to Add an existing host back to the vCenter getting error "Cannot complete login due to an incorrect user name or password"

Unable to add host due to incorrect username or password.