Password remediation for ESXi and vCenter accounts fails with handshake failure in SDDC Manager

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• In the SDDC Manager UI, ESXi and vCenter service/root accounts are in a disconnected state.
• Attempts to remediate or update the passwords for these accounts fail. The task fails with the following error message:
  

  error: handshake timed out after 10000ms

• Error in /var/log/vmware/vcf/operationsmanager/operationsmanager.log
  

  YYYY-MM-DDTHH:MM:SS ERROR [vcf_om,################################,####] [c.v.e.s.c.c.v.vsphere.VsphereClient,om-exec-7] Failed to connect to https://<HOST_FQDN>:443/sdk
  com.vmware.vim.vmomi.client.exception.ConnectionException: https://<HOST_FQDN>:443/sdk invocation failed with "org.apache.http.conn.ConnectTimeoutException: Connect to <HOST_FQDN>:443 [<HOST_FQDN>/<HOST_IP_ADDRESS>] failed: Read timed out"
          at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:265)
  

Environment

VMware Cloud Foundation 5.2.2

Cause

An intervening firewall or network security appliance blocks required TCP port 443 traffic between the SDDC Manager, vCenter Server, and the target ESXi hosts. SDDC Manager relies on secure SSL/TLS sessions over port 443 to authenticate with host agents and execute password rotation tasks. If this traffic is dropped, the TLS handshake cannot initialize or complete, resulting in a handshake failure that prevents the accounts from synchronizing.

Resolution

1. Validate network routing and connectivity between the SDDC Manager appliance, vCenter Server, and the management IP addresses of the affected ESXi hosts.

   • From SDDC Manager: Use the curl command to test connectivity to the vCenter Server and ESXi hosts.
     

     curl -v telnet://<VCENTER_IP_OR_FQDN>:<Port_Number>
     curl -v telnet://<ESXI_MGMT_IP_OR_FQDN>:<Port_Number>

      

   • From the ESXi host(s): Use the netcat command (nc) to test connectivity to the SDDC Manager and vCenter Server.
     

     nc -z <SDDC_MANAGER_IP_OR_FQDN> <Port_Number>
     nc -z <VCENTER_IP_OR_FQDN> <Port_Number>

      

   • From the vCenter Server: Use the curl command to test connectivity to the SDDC Manager and ESXi hosts.

     curl -v telnet://<SDDC_MANAGER_IP_OR_FQDN>:<Port_Number>
     curl -v telnet://<ESXI_MGMT_IP_OR_FQDN>:<Port_Number>

      

     The following KB article can be referred to review the network port requirements for VMware Cloud Foundation:
     https://knowledge.broadcom.com/external/article/316756/network-port-requirements-for-vmware-clo.html

2. Review the physical and virtual firewall configurations along the network path.

3. Ensure that bidirectional communication over TCP port 443 is explicitly allowed.

4. Modify or remove any firewall rules that are dropping or rejecting the HTTPS/443 packets.

5. After network communication is restored, navigate to the SDDC Manager UI and re-initiate the password remediation task for the disconnected accounts.

Additional Information

Firewall Rules for a  Protected SDDC:
https://techdocs.broadcom.com/de/de/vmware-cis/private-ai/foundation-with-nvidia/5-2/cyber-recovery-connector-firewall-rules-for-a-protected-sddc.html