VMware Update Manager Scan/Remediation fails on vSAN Witness node ESXi Hosts

Products

• VMware vSAN

Issue/Introduction

Symptoms:
•  VMware Update Manager Scan/Remediation fails on vSAN Witness node ESXI Hosts

• In vmware-vum-server-log4cpp.log log of Update manager, you see entries similar to:

[2018-09-07 03:00:07:226 'SingleHostScanTask.SingleHostScanTask{1005}' 139824396990208 ERROR] [singleHostScanTask, 402] SingleHostScan caught exception: 8 with code: 129
[2018-09-07 03:00:07:229 'SingleHostScanTask.SingleHostScanTask{1005}' 139824396990208 ERROR] [vciTaskBase, 557] Task execution has failed: integrity.fault.EsxupdateErrorVibFormatFault
  

• In esxupdate.log log of vSAN Witness node ESXi host, you see entries similar to:

2018-09-07T03:00:41Z esxupdate: 69840: esxupdate: ERROR: An esxupdate error exception was caught:
2018-09-07T03:00:41Z esxupdate: 69840: esxupdate: ERROR: Traceback (most recent call last):
2018-09-07T03:00:41Z esxupdate: 69840: esxupdate: ERROR: File "/build/mts/release/bora-8935087/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Metadata.py", line 75, in ReadMetadataZip
2018-09-07T03:00:41Z esxupdate: 69840: esxupdate: ERROR: File "/build/mts/release/bora-8935087/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/zipfile.py", line 1235, in read
2018-09-07T03:00:41Z esxupdate: 69840: esxupdate: ERROR: File "/build/mts/release/bora-8935087/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/zipfile.py", line 830, in read
2018-09-07T03:00:41Z esxupdate: 69840: esxupdate: ERROR: File "/build/mts/release/bora-8935087/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/zipfile.py", line 920, in _read1
2018-09-07T03:00:41Z esxupdate: 69840: esxupdate: ERROR: MemoryError
 

• In the VC GUI  Host and Cluster view :  Host > Monitor > Task & Events > Tasks shows the following error 

Status: The host returns esxupdate error code:99. An unhandled exception was encountered.Check the Update Manager log files and esxupdate log files for more details

Environment

VMware vSAN 6.x

Cause

This issue occurs due to smaller hostd resource group in vSAN Witness node ESXi hosts compared to normal ESXi  hosts to process large number of metadata files during host scan or remediation.

Resolution

Upgrade vCenter to 7.0U1 or higher

Additional Information

Using vSphere Update Manager to upgrade hosts in parallel might result in the witness host being upgraded in parallel with one of the data hosts in a stretched cluster / 2-Node configurations. 
To avoid upgrade problems, do not configure VMware Update Manager to upgrade a witness host in parallel with the data hosts in a stretched cluster. 
Upgrade the witness host after all data hosts have been successfully upgraded and have exited maintenance mode.