"VMware vSphere Update Manager had an unknown error" while scanning host for upgrade on 6.7 U3

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

This article is to identify and resolve the "Host Upgrade" Compliance scan failure with error "VMware vSphere Update Manager had an unknown error".

Symptoms:

• "Host Upgrade" Compliance scan on 6.7U3 Update Manager fails with error "VMware vSphere Update Manager had an unknown error".
• You will see similar log entries in /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-log4cpp.log log file as mentioned below,

vmware-vum-server-log4cpp.log:
[2019-10-08 11:35:23:332 'JobDispatcher' 139791892846336 INFO]  [JobDispatcher, 1860] Scheduling task SingleHostUpgradeScanTask{16} that imposes 1 for ESX: host-7267 new load: 1 that imposes 1 for VMware vSphere Update Manager new load: 1
[2019-10-08 11:35:23:394 'SingleHostUpgradeScanTask.SingleHostUpgradeScanTask{16}' 139791892580096 INFO]  [singleHostUpgradeScanTask, 338] Scanning host esxi1.vmwarelab.com against upgrade VMware ESXi 6.7.0 Update 2 (profile name = DELL_ESXi-6.7u2-13006603_Customized-A00-NO--MLX-lsi_mr3_7.705.10)
[2019-10-08 11:35:27:559 'AgentDeploy' 139791892580096 INFO]  [agentDeploy, 247] Agent installed
[2019-10-08 11:35:27:560 'SingleHostUpgradeScanTask.SingleHostUpgradeScanTask{16}' 139791892580096 ERROR]  [singleHostUpgradeScanTask, 277] Error while scanning: Connection reset by peer: The connection is terminated by the remote end with a reset packet. Usually, this is a sign of a network problem,  timeout, or service overload.

Note:

1. Above VUM log shows that the VUM received a reset from the ESXi while performing the Scan.
2. In ESXi host esxupdate.log will not have any error.

• In such scenario, Packet capture on both the VCSA and ESXi will show more data to understand the cause of failure.

Example: (ESXi packet capture sample)
3584    2019-10-22 20:25:18.924715    10.19.14.6    10.19.14.20    TCP    66    58940 → 80 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=256
3392    2019-10-22 20:25:18.924729    10.19.14.20    10.19.14.6    TCP    66    80 → 58940 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=512 SACK_PERM=1
3585    2019-10-22 20:25:18.924954    10.19.14.6    10.19.14.20    TCP    60    58940 → 80 [ACK] Seq=1 Ack=1 Win=29440 Len=0

3586    2019-10-22 20:25:18.925990    10.19.14.6    10.19.14.20    TCP    60    58940 → 80 [RST, ACK] Seq=1 Ack=1 Win=29440 Len=0

• Packet capture on ESXi shows that vCenter connected to ESXi on TCP port 80 and after TCP 3 way handshake vCenter sent a "Reset" packet to ESXi.

Example: (vCenter Server packet capture sample)

4029    2019-10-22 20:25:18.924969    10.19.14.6    10.19.14.20    TCP    66    58940 → 80 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=256
4030    2019-10-22 20:25:18.925351    10.19.14.20    10.19.14.6    TCP    66    80 → 58940 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=512 SACK_PERM=1
4031    2019-10-22 20:25:18.925386    10.19.14.6    10.19.14.20    TCP    54    58940 → 80 [ACK] Seq=1 Ack=1 Win=29440 Len=0

4033    2019-10-22 20:25:18.926547    10.19.14.20    10.19.14.6    TCP    60    80 → 58940 [RST, ACK] Seq=1 Ack=140 Win=58880 Len=0

• Packet capture on vCenter shows that vCenter connected to ESXi on TCP port 80 and after TCP 3 way handshake ESXi sent a "Reset" packet to vCenter.
• Such behavior can be seen if there is a Firewall acting as an intermediate device closing session (Sending Reset) on both ends.

Note:

1. The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
2. Above issue occurs for a specific set of ESXi which are behind Firewall.
3. Above issue is not seen for "Patch Scan"

Environment

VMware vSphere ESXi 6.7
VMware vSphere Update Manager 6.7.x

Cause

Firewall between ESXi and vCenter has a specific rule to block data transfer on TCP port 80.

Resolution

To resolve this issue remove the Firewall policy between the vCenter and ESXi on Port 80.