vSphere Lifecycle Manager or vSphere Update Manager compliance checks fail with the error: 'VMware vSphere Lifecycle Manager encountered an unknown error. Please review the events and log files for more information,'

Products

• VMware vCenter Server

Issue/Introduction

Compliance check tasks in vSphere Lifecycle Manager are failing for hosts that are managed using baselines.

The vSphere Update Manager logs, /var/log/vmware/vmware-updatemgr/vmware-vum-server.log, may show the below events: 

yyyy-mm-ddThh:mm:ss.Z info vmware-vum-server[44328] [Originator@6876 sub=ServerConnection] [serverConnection 93] connecting vua on port 80
[...]
yyyy-mm-ddThh:mm:ss.Z warning vmware-vum-server[44912] [Originator@6876 sub=IO.Connection] Failed to connect; <io_obj p:0x00007f5dbccb3998, h:30, <TCP '##.##.##.## : 34598'>, <TCP '##.##.##.## : 80'>>, e: 110(Connection timed out), duration: 129636msec
yyyy-mm-ddThh:mm:ss.Z warning vmware-vum-server[44912] [Originator@6876 sub=HttpConnectionPool-000000] Failed to get pooled connection; <cs p:00007f5dc80a3a50, TCP:##.##.##.##:80>, (null), duration: 129637msec, N7Vmacore15SystemExceptionE(Connection timed out)

yyyy-mm-ddThh:mm:ss.Z info vmware-vum-server[27415] [Originator@6876 sub=ServerConnection] [serverConnection 94] connecting vua on port 80
yyyy-mm-ddThh:mm:ss.Zwarning vmware-vum-server[27429] [Originator@6876 sub=IO.Connection] Failed to SSL handshake; SSL(<io_obj p:0x00007f75d404cbc0, h:49, <TCP '##.##.##.## : 38866'>, <TCP '##.##.##.## : 80'>>), e: 104(Connection reset by peer), duration: 1msec
yyyy-mm-ddThh:mm:ss.Zwarning vmware-vum-server[27429] [Originator@6876 sub=HttpConnectionPool-000000] Failed to get pooled connection; <cs p:00007f759c0023d0, TCP:vc_fqdn:80>, SSL(<io_obj p:0x00007f75d404cbc0, h:49, <TCP '##.##.##.## : 38866'>, <TCP '##.##.##.## : 80'>>), duration: 3msec, N7Vmacore15SystemExceptionE(Connection reset by peer: The connection is terminated by the remote end with a reset packet. Usually, this is a sign of a network problem,  timeout, or service overload.)

Environment

• VMware vCenter Server 8.0.x
• VMware vCenter Server 7.0.x

Cause

• When executing scan or remediation tasks associated with an Upgrade baseline, vSphere Lifecycle Manager (vLCM) deploys a lightweight agent called the VMware Update Agent (VUA) to the ESXi host.
• By default, vLCM communicates with the VUA over TCP port 80. If this port is blocked or inaccessible, the communication will fail, resulting in task failure.
• This error may also occur if the download token is not entitled to download the latest patch versions of ESXi and vCenter Server.
  
  

 

Resolution

To resolve the issue,

• Allow port 80 from vCenter Server to the ESXi hosts. For more information see, VMware Ports and Protocols.
  
  

  

• Validate and update the Download Token in vCenter Server. See KB, Error: A general system error occurred: Failed to download VIB(s): Error: HTTP Error Code: 403, vLCM fails to download the ESXi patches and images from online repositories

If port 80 cannot be enabled due to security policies, work around by configuring vLCM to communicate with the VUA agent over an alternate port.

To modify the port used by vLCM:

• SSH into the vCenter Server using root credentials.
• Stop the vLCM Update Manager service:
  • service-control --stop updatemgr
• Back up the vLCM configuration file (vci-integrity.xml):
  • cp /usr/lib/vmware-updatemgr/bin/vci-integrity.xml /usr/lib/vmware-updatemgr/bin/vci-integrity.bak
• Edit the configuration file:
  • vi /usr/lib/vmware-updatemgr/bin/vci-integrity.xml
• Locate and update the <upgradePort> value from 80 to an alternative open port (e.g., 443):
  • <vuaClient>
    <upgradePort>443</upgradePort>
</vuaClient>
• Save the file and restart the Update Manager service:
  • service-control --start updatemgr
• Once the service is restarted, log in to vCenter and reattempt the Check Compliance operation.

Additional Information

Network Port Requirements -

• VMware Ports and Protocols - https://ports.broadcom.com/