vSphere Lifecycle Manager remediation fails with "VMware vSphere Lifecycle Manager encountered an unknown error. Please review the events and log files for more information".

Products

• VMware vCenter Server

Issue/Introduction

• Remediation task in vSphere Lifecycle Manager fails for hosts that are managed using baselines
  
  

• Communication over port 9084,80 and 443 is successful between vCenter and ESXi.
  
  
• In /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log : 
  
  

  <YYYY-MM-DD>T<time> warning vmware-vum-server[2411297] [Originator@6876 sub=IO.Connection] Failed to SSL handshake; SSL(<io_obj p:0x00007f8f98001cd0, h:-1, <TCP '#.#.#.# : 35504'>, <TCP '#.#.#.# : 80'>>), e: 125(Operation canceled), duration: 120248msec
  
  <YYYY-MM-DD>T<time> warning vmware-vum-server[2411297] [Originator@6876 sub=HttpConnectionPool-000000] Failed to get pooled connection; <cs p:00007f8f7c02ac90, TCP:#.#.#.#:80>, SSL(<io_obj p:0x00007f8f98001cd0, h:-1, <TCP '#.#.#.# : 35504'>, <TCP '#.#.#.# : 80'>>), duration: 120288msec, N7Vmacore3Ssl28SSLHandshakeTimeoutExceptionE(SSL Exception: Operation was canceled)^M

Environment

VMware vSphere ESXi 7.x
VMware vSphere vCenter 7.x

VMware vSphere ESXi 8.x
VMware vSphere vCenter 8.x

Cause

The VMware Update Agent (VUA) establishes a tunnel over Port 80 to facilitate host scans and updates. While the initial connection is HTTP-based, the tunnel subsequently carries encapsulated binary TLS data.

This issue typically occurs when a network firewall with Deep Packet Inspection (DPI) or an Intrusion Prevention System (IPS) is positioned between the vCenter Server and the ESXi hosts.

When firewalls fail to recognize this protocol transition and continue to apply strict text-based HTTP validation to the binary stream. Because TLS data is randomized, the hex value 0x0D (Carriage Return) frequently appears without a trailing 0x0A (Line Feed). The firewall flags this as a malformed HTTP header and misclassifies the traffic as an "HTTP Request Smuggling" exploit. Consequently, the firewall silently drops the frames, leading to retransmissions and an eventually timed-out connection.

Resolution

Reconfigure vCenter to route VUA traffic over Port 443 (HTTPS), which bypasses the firewall's text-based HTTP inspection. To do so, follow the steps outlined in KB

vsphere-lifecycle-manager-or-vsphere-upd.html

 

Additional Information

"VMware vSphere Lifecycle Manager encountered an unknown error. Please review the events and log files for more information" when scanning host for upgrade/update on 7.0.x or 8.0.x