ESXi host fails to add to vCenter with error "Cannot complete login due to an incorrect user name or password"

Products

• VMware vCenter Server

Issue/Introduction

Symptoms -

While trying to add the ESXi host to vCenter, it fails with "Cannot complete login due to an incorrect user name or password"

ESXi root password is verified to be correct

Logs show below entries -

vpxd.log

XXXX-XX-XXTXX:XX:XX.XXXZ error vpxd[36138] [Originator@6876 sub=IO.Http opID=xxxxxxxxx:xxxxx] User agent failed to send request; SSL(<io_obj p:0x00007fa504071018, h:-1, <TCP 'X.X.X.X:35670'>, <TCP 'Y.Y.Y.Y:443'>>), N7Vmacore4Http24MalformedHeaderExceptionE(Server closed connection after 0 response bytes read)
--> [context]zKq7AVECAQAAAH1IQAEPdnB4ZAAA7tg3bGlidm1hY29yZS5zbwAAp3gsACVsLQA64DIAPKcwAEqqMABW6TAAaiUtAFQEIwCgqCMAp0kjAE+fIwCsZDcBh38AbGlicHRocmVhZC5zby4wAAIvNg9saWJjLnNvLjYA[/context]

XXXX-XX-XXTXX:XX:XX.XXXZ error vpxd[35938] [Originator@6876 sub=TrustedInfrastructure.HostConfig opID=xxxxx:xxxx] [StsUploader] Failed to set STS certificates to host 'Y.Y.Y.Y'. Error:

--> Error:
--> system_error
--> Messages:
--> vapi.send.failed<Send of frame failed: N7Vmacore4Http24MalformedHeaderExceptionE(Server closed connection after 0 response bytes read)
--> [context]xxxxxxx[/context]>
-->
XXXX-XX-XXTXX:XX:XX.XXXZ error vpxd[35938] [Originator@6876 sub=InvtHost opID=xxxxx:xxxx] Failed to reconnect to cleanup before host removal: N3Vim5Fault12InvalidLogin9ExceptionE(Fault c
ause: vim.fault.InvalidLogin

Environment

vSphere ESXi 7.0

Cause

This indicates a possible network issue between vCenter and ESXi. 

Resolution

Please engage your Network team and recheck the MTU/MSS values. 

If all seems correct and working, please collect below information and open a Support Request with Broadcom-

1. On VC, start tcpdump to capture TCP traffic between VC and the "problem" ESXi host:

tcpdump -i eth0 -s 512 -w /var/core/vcpackets.pcap -W 50 -C 20 "host <hostIP> and port 443"

2. On the "problem" ESXi host, start pktcap-uw to capture TCP traffic on both vmk and its corresponding vmknic:

pktcap-uw --vmk vmk0 --dir 2 --tcpport 443 -s 512 -o /var/run/log/vmk0.pcap

pktcap-uw -K --uplink <vmnic0> --dir 2 --tcpport 443 -s 512 -o /var/run/log/vmnic0.pcap

3. Reproduce the failure to add the ESXi host to VC.

4. Stop network captures on VC and the ESXi host.

5. Collect and upload support bundles (with network captures) from VC and ESXi host to the Support Request.