Unable to install vmtools or ESXi patch using lifecycle manager fails with: An error occured during host configuration /usr/sbin/esxupdate returned with exit status :32 in vSphere 7.x

Products

VMware vSphere ESX 7.x VMware vCenter Server VMware vCenter Server 7.0

Issue/Introduction

Unable to install VMtools or ESXi patch via lifecycle manager.
No issues were found while patching via the command line.
Update initiated by new baseline with "automatically updates" unchecked also fails.
Error received on the UI: An error occurred during host configuration
/usr/sbin/esxupdate returned with exit status: 32

Log location: /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

The patching fails with an error in the logs stating it failed to install the NSX vibs:



YYYY-MM-DDTHH:MM:SS.488+08:00 info vmware-vum-server[177403] [Originator@6876 sub=HostUpdateDepotManager] [vibDownloaderImpl 361] Downloaded and validated vibs successfully
YYYY-MM-DDTHH:MM:SS.488+08:00 info vmware-vum-server[177403] [Originator@6876 sub=HostUpdateDepotManager] [patchDepotManager 692] Vib file: /storage/updatemgr/patch-store/hostupdate/vmw/vib20/esx-nsxv/VMware_bootbank_esx-nsxv_6.7.0-0.0.17493714.vib is already downloaded (and being copied).

YYYY-MM-DDTHH:MM:SS.637+08:00 info vmware-vum-server[422112] [Originator@6876 sub=HostUpdateDepotManager] [scanHostxxx] List of updates for installation/staging on host: ### (entity: host-###):
--> -----------------------------------------------
--> (VMware_bootbank_esx-nsxv_6.7.0-0.0.17493714, VMware_locker_tools-light_12.3.5.22544099-22545121)
--> ------------------------------------------------
YYYY-MM-DDTHH:MM:SS.637+08:00 info vmware-vum-server[422112] [Originator@6876 sub=SingleHostRemediateTask.SingleHostRemediateTask{375}] [singleHostRemediateTaskBase 392] 
--> ------------------------------------------------
--> (VMware_bootbank_esx-nsxv_6.7.0-0.0.17493714, VMware_locker_tools-light_12.3.5.22544099-22545121)
--> ------------------------------------------------

The baseline VM Tools does not contain NSX-V

 
YYYY-MM-DDTHH:MM:SS.282+08:00 info vmware-vum-server[09460] [Originator@6876 sub=BaselineMgr] [baselineMgr 5497] Baseline 'VMWare Tools 12.3.0' with ID '21' has 'TOOLS-22236987', updates

However, the EAM baselines contain NSX-V vib in the updates list:

YYYY-MM-DDTHH:MM:SS.308+08:00 info vmware-vum-server[09460] [Originator@6876 sub=BaselineMgr] [baselineMgr 5497] Baseline 'ESX Agent Manager 
:: 56d8c027-####-####-####-########ad1' with ID '10' has 'nsxv-datapath-6.7-17493714', 'nsxv-datapath-6.5-17493714', updates.

YYYY-MM-DDTHH:MM:SS.350+08:00 info vmware-vum-server[09460] [Originator@6876 sub=BaselineMgr] [baselineMgr 5497] Baseline 'ESX Agent Manager 
:: bfd62c04-####-####-####-########09e' with ID '8' has 'nsxv-datapath-6.7-17493714', 'nsxv-datapath-6.5-17493714', updates.

YYYY-MM-DDTHH:MM:SS.385+08:00 info vmware-vum-server[09460] [Originator@6876 sub=BaselineMgr] [baselineMgr 5497] Baseline 'ESX Agent Manager 
:: 82035b01-####-####-####-########808' with ID '9' has 'nsxv-datapath-6.7-17493714', 'nsxv-datapath-6.5-17493714', updates.

Environment

Environment has NSX installed

Cause

vSphere ESX Agent Manager (EAM) may stage and install VIBs through VUM depending on the VIB type.
There are legacy VIBs that are installed through VUM, and there are vLCM VIBs that are installed through vLCM.

Eam.log

YYYY-MM-DDTHH:MM:SS.547Z |  INFO | vim-monitor | ExecutorUtils.java | 36 | JOB SUBMITED: [#19808345] StageAgencyJob(Agency ID: bfd62c04-####-####-####-########09e, Depots: [https://x.x.x.x/bin/vdn/vibs-6.4.10/6.5-17493714/vxlan.zip, https://x.x.x.x/bin/vdn/vibs-6.4.10/6.7-17493714/vxlan.zip], Scope: [ClusterComputeResource:domain-c2307], PM available: false, VUM available: true)
YYY-MM-DDTHH:MM:SS.576Z |  INFO | agency-1 | AuditedJob.java | 106 | JOB STARTED: [#19808345] StageAgencyJob(Agency ID: bfd62c04-####-####-####-########09e, Depots: [https://x.x.x.x/bin/vdn/vibs-6.4.10/6.5-17493714/vxlan.zip, https://x.x.x.x/bin/vdn/vibs-6.4.10/6.7-17493714/vxlan.zip], Scope: [ClusterComputeResource:domain-c2307], PM available: false, VUM available: true)

YYYY-MM-DDTHH:MM:SS.950Z | ERROR | host-292524-0 | VibJob.java | 300 | Cannot access vib package at https://x.x.x.x/bin/vdn/vibs-6.4.10/6.7-17493714/vxlan.zip
com.vmware.eam.security.trust.CannotConnect: Cannot open connection.

YYYY-MM-DDTHH:MM:SS.197Z |  INFO | agency-1 | StageAgencyJob.java | 144 | [19808345:6e2079cf85b23eba] Staging agency finished: JobResult[parsedDepots=Result[value=null, exception=com.vmware.eam.exception.CannotHostUrlException: URL:https://x.x.x.x/bin/vdn/vibs-6.4.10/6.5-17493714/vxlan.zip cant be accessed], vumRes=VumResult[operation=NO_OP, uploadedVibs=Result[value=[], exception=null], baselineRecRes=Result[value=null, exception=null]], pmRes=PmResult[operation=NO_OP, clustersRes=[], depotsRes=Result[value=null, exception=null]]]
YYYY-MM-DDTHH:MM:SS.197Z |  INFO | agency-1 | AuditedJob.java | 106 | JOB COMPLETED: [#19808345] StageAgencyJob(Agency ID: bfd62c04-####-####-####-########09e, Depots: [https://x.x.x.x/bin/vdn/vibs-6.4.10/6.5-17493714/vxlan.zip, https://x.x.x.x/bin/vdn/vibs-6.4.10/6.7-17493714/vxlan.zip], Scope: [ClusterComputeResource:domain-c2307], PM available: false, VUM available: true)

YYYY-MM-DDTHH:MM:SS.196Z |  INFO | manage-depots-LRO | Workflow.java | 121 | [LoadDepot:96e7463f47bae98] FAILED
com.vmware.eam.exception.CannotHostUrlException: URL:https://x.x.x.x/bin/vdn/vibs-6.4.10/6.5-17493714/vxlan.zip cant be accessed
 at com.vmware.eam.job.UrlHostJob.call(UrlHostJob.java:107) ~[eam-server.jar:?]
 at com.vmware.eam.job.UrlHostJob.call(UrlHostJob.java:33) ~[eam-server.jar:?]
 at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:58) ~[eam-server.jar:?]
 at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:55) [eam-server.jar:?]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_362]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_362]
 at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
Caused by: com.vmware.eam.security.trust.CannotConnect: Cannot open connection.
 at com.vmware.eam.security.trust.impl.TrustedLeaf.get(TrustedLeaf.java:141) ~[eam-server.jar:?]
 at com.vmware.eam.envoy.route.impl.RouteProvider.provide(RouteProvider.java:193) ~[eam-server.jar:?]
 at com.vmware.eam.envoy.route.impl.RouteProvider.provide(RouteProvider.java:299) ~[eam-server.jar:?]
 at com.vmware.eam.job.UrlHostJob.call(UrlHostJob.java:96) ~[eam-server.jar:?]
 ... 6 more
Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)
 at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_362]

Resolution

Remove the EAM baseline.
Remove the EAM agency.
Connect to VCDB and check the table:

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

SELECT * FROM VPX_EXT_DATA WHERE ext_id = 'com.vmware.vim.eam';

The output is similar to the code below and shows the baselines, which are seen on the vCenter from the old NSX-V manager:

ESX Agent Manager :: 56d8c027-####-####-####-########ad1
ESX Agent Manager :: bfd62c04-####-####-####-########09e
ESX Agent Manager :: 82035b01-####-####-####-########808

Then follow the steps below to resolve the issue:

Take an offline snapshot of the vCenter server appliance before performing the below steps.
Stop the following services:
service-control --stop vmware-vpxd
service-control --stop eam
Delete the agent details from the VCDB:

Example:
delete from VPX_EXT_DATA where ext_id = 'com.vmware.vim.eam' and data_key like '%56d8c027-####-####-####-########ad1%';
Restart vCenter services and validate the lifecycle manager to see if the old baseline of EAM is deleted.
service-control --start vmware-vpxd
service-control --start eam
Attach the VMware tools baseline and remediate the host.