Snapshot backup for VMs in vCenter server are failing due to HTTP Connection has timed out while waiting for further requests

Products

• VMware vCenter Server

Issue/Introduction

• Snapshot backups were failing with error

  HTTP Connection has timed out while waiting for further requests.

   

• Error in /var/log/vmware/vpxd/vpxd.log

  info vpxd[######] [Originator@6876 sub=HttpSvc.HTTPService.HttpConnection] HTTP Connection has timed out while waiting for further requests; <io_obj p:0x00######be08, h:-1, <UNIX '/var/run/vpxd/vpxd-webserver-pipe'>, <UNIX ''>>, N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <io_obj p:0x0######be08, h:-1, <UNIX '/var/run/vpxd/vpxd-webserver-pipe'>, <UNIX ''>>, duration: 00:00:47.571368 (hh:mm:ss.us))
  info vpxd[[######] [Originator@6876 sub=VapiEndpoint.HTTPService.HttpConnection] HTTP Connection has timed out while waiting for further requests; <io_obj p:0x00007fe8f0116db0, h:-1, <TCP '127.0.0.1 : 8093'>, <TCP '127.0.0.1 : 36420'>>, N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <io_obj p:0x00######db0, h:-1, <TCP '127.0.0.1 : 8093'>, <TCP '127.0.0.1 : 36420'>>, duration: 00:00:48.764768 (hh:mm:ss.us))

   

• In /var/log/vmware/envoy/envoy.log shows the below

  info envoy[3038] [Originator@6876 sub=http] [Tags: "ConnectionId":"4005981"] idle timeout: closing connection TCP<10.109.##.##:39434, 10.102.##.##:443>
  info envoy[3041] [Originator@6876 sub=http] [Tags: "ConnectionId":"4006508"] idle timeout: closing connection TCP<10.100.##.##:38340, 10.102.##.##:443>
  info envoy[3040] [Originator@6876 sub=http] [Tags: "ConnectionId":"4006935"] idle timeout: closing connection TCP<10.101.##.##:32770, 10.102.##.##:443>
  info envoy[3038] [Originator@6876 sub=http] [Tags: "ConnectionId":"4007038"] idle timeout: closing connection TCP<10.113.##.##:33776, 10.102.##.##:443>

Cause

High number of idle/stale HTTP sessions overloading vCenter's HTTP service.

Resolution

1. SSH to vCenter server as root user
2. Browse to /var/log/vmware/envoy

   cd /var/log/vmware/envoy

3. Run the below command to list the IPs with HTTP sessions to the vCenter

   grep '.' envoy-access-* | cut -d ' ' -f 18 | cut -d ':' -f 1 | sort | uniq -c | sort -nr | head

   Sample output

    165077 10.102.##.##
    111828 10.109.##.##
     86028 10.116.##.##
     84425 10.103.##.##
     80357 10.103.##.##
     80079 10.102.##.##
     74103 10.118.##.##
     72999 10.102.##.##
     72478 10.109.##.##
     71483 10.113.##.##

4. Identify the source IPs from Step # 4 and investigate from Client end to close the connections.
5. You can also see the downstream connections that are timing out as a consequence with the below command, you can see which IPs are timing out as a consequence of high number of sessions from the above list.
   

   grep "idle timeout" envoy*.log | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort | uniq -c | sort -nr | head 
    2024 10.198.##.##
      1349 10.198.##.##
       332 10.198.##.##
       308 10.198.##.##