Management vCenter Server Login Slowness and Backup Failures Due to Network Packet Loss

Products

• VMware vCenter Server
• VMware Cloud Foundation

Issue/Introduction

In a VMware Cloud Foundation environment, Management vCenter Server web client logins experience severe slowness or failure, requiring up to 10 minutes to authenticate.

Simultaneously, third-party server backup operations targeting the vCenter appliance fail to complete.

Backup Server throws error;

Failed to connect to <vCenter FQDN> . Error: SOAP 1.1 fault SOAP-ENV:Server[no subcode] 'Timeout' Detail: connect failed in tcp_connect()

 

Environment

VMware Cloud Foundation 9.x

VMware vCenter Server

Cause

Intermittent network packet loss and TCP connection timeouts occurring at the Top of Rack (TOR) switch level.

During connection verification from the backup node to the vCenter appliance over HTTPS (port 443), curl execution results in a connection timeout:

$ curl -v -k --connect-timeout 30 "https://<vCenter FQDN>/sdk"
*   Trying ##.###.##.###:443...
* Connection timed out after 30001 milliseconds
* Closing connection 0
curl: (28) Connection timed out after 30001 milliseconds

Resolution

To resolve this issue, perform the following steps to clear network degradation on the Top of Rack (TOR) switch:

1. Inspect the Top of Rack (TOR) switch interface counters, error statistics, and physical link metrics for dropped frames or buffer errors.
2. Perform a planned reboot of the Top of Rack (TOR) switch to clear hardware buffer congestion and restore frame throughput.
3. Validate network layer stability from the backup node to the vCenter Server over TCP port 443 using curl and nping:
   1. curl -v -k --connect-timeout 30 "https://<vCenter FQDN>/sdk"
nping <vCenter FQDN> -p 443
4. Confirm 0% packet loss and successful TCP handshakes.
5. Verify Management vCenter Server UI login responsiveness.
6. Initiate backup jobs to confirm successful execution.