Unable to install vmtools or Esxi patch using lifecycle manager fails with: An error occured during host configuration /usr/sbin/esxupdate returned with exit status :32 in vSphere 7.x

Products

• VMware vSphere ESX 7.x
• VMware vCenter Server
• VMware vCenter Server 7.0

Issue/Introduction

1. Unable to install VMtools or ESXi patch via lifecycle manager.

2. No issues found while patching via command line.

3. Update initiated by new baseline with "automatically updates" unchecked also fails.

4.Error received on the UI :An error occured during host configuration
/usr/sbin/esxupdate returned with exit status :32

Environment

Infra has NSX installed.

Cause

 

Log location: /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

The patching  failing with error in logs stating its failing to install the nsx vibs. 

2023-11-03T14:57:59.488+08:00 info vmware-vum-server[177403] [Originator@6876 sub=HostUpdateDepotManager] [vibDownloaderImpl 361] Downloaded and validated vibs successfully
2023-11-03T14:57:59.488+08:00 info vmware-vum-server[177403] [Originator@6876 sub=HostUpdateDepotManager] [patchDepotManager 692] Vib file: /storage/updatemgr/patch-store/hostupdate/vmw/vib20/esx-nsxv/VMware_bootbank_esx-nsxv_6.7.0-0.0.17493714.vibis already downloaded (and being copied).

2023-12-20T11:03:25.637+08:00 info vmware-vum-server[422112] [Originator@6876 sub=HostUpdateDepotManager] [scanHostxxx] List of updates for installation/staging on host: xxx (entity: host-xxx):
--> -----------------------------------------------
--> (VMware_bootbank_esx-nsxv_6.7.0-0.0.17493714, VMware_locker_tools-light_12.3.5.22544099-22545121)
--> ------------------------------------------------
2023-12-20T11:03:25.637+08:00 info vmware-vum-server[422112] [Originator@6876 sub=SingleHostRemediateTask.SingleHostRemediateTask{375}] [singleHostRemediateTaskBase 392] 
--> ------------------------------------------------
--> (VMware_bootbank_esx-nsxv_6.7.0-0.0.17493714, VMware_locker_tools-light_12.3.5.22544099-22545121)
--> ------------------------------------------------

 

 

The baseline VM Tools doesn't contain nsxv

 
2023-10-27T12:24:14.282+08:00 info vmware-vum-server[09460] [Originator@6876 sub=BaselineMgr] [baselineMgr 5497] Baseline 'VMWare Tools 
12.3.0' with ID '21' has 'TOOLS-22236987', updates

 

However the EAM baselines does seem to contain nsxv vib in the updates list

 

2023-10-27T12:24:14.308+08:00 info vmware-vum-server[09460] [Originator@6876 sub=BaselineMgr] [baselineMgr 5497] Baseline 'ESX Agent Manager 
:: 56d8c027-f15f-497f-9d89-c6bcb1f13ad1' with ID '10' has 'nsxv-datapath-6.7-17493714', 'nsxv-datapath-6.5-17493714', updates.

2023-10-27T12:24:14.350+08:00 info vmware-vum-server[09460] [Originator@6876 sub=BaselineMgr] [baselineMgr 5497] Baseline 'ESX Agent Manager 
:: bfd62c04-4cd6-4e07-962d-546687f6a09e' with ID '8' has 'nsxv-datapath-6.7-17493714', 'nsxv-datapath-6.5-17493714', updates.

2023-10-27T12:24:14.385+08:00 info vmware-vum-server[09460] [Originator@6876 sub=BaselineMgr] [baselineMgr 5497] Baseline 'ESX Agent Manager 
:: 82035b01-5c0e-4084-a643-1945cfb34808' with ID '9' has 'nsxv-datapath-6.7-17493714', 'nsxv-datapath-6.5-17493714', updates.

 

Cause: 

 

EAM may stage and install VIBs through VUM depending on the VIB type
There are legacy VIBs that are installed through VUM and there are vLCM VIBs that are installed through vLCM

 

 

Eam.log 

 

2023-11-03T04:45:29.547Z |  INFO | vim-monitor | ExecutorUtils.java | 36 | JOB SUBMITED: [#19808345] StageAgencyJob(Agency ID: bfd62c04-4cd6-4e07-962d-546687f6a09e, Depots: [https://x.x.x.x/bin/vdn/vibs-6.4.10/6.5-17493714/vxlan.zip, https://x.x.x.x/bin/vdn/vibs-6.4.10/6.7-17493714/vxlan.zip], Scope: [ClusterComputeResource:domain-c2307], PM available: false, VUM available: true)
2023-11-03T04:45:29.576Z |  INFO | agency-1 | AuditedJob.java | 106 | JOB STARTED: [#19808345] StageAgencyJob(Agency ID: bfd62c04-4cd6-4e07-962d-546687f6a09e, Depots: [https://x.x.x.x/bin/vdn/vibs-6.4.10/6.5-17493714/vxlan.zip, https://x.x.x.x/bin/vdn/vibs-6.4.10/6.7-17493714/vxlan.zip], Scope: [ClusterComputeResource:domain-c2307], PM available: false, VUM available: true)

2023-11-03T04:45:47.950Z | ERROR | host-292524-0 | VibJob.java | 300 | Cannot access vib package at https://x.x.x.x/bin/vdn/vibs-6.4.10/6.7-17493714/vxlan.zip
com.vmware.eam.security.trust.CannotConnect: Cannot open connection.

2023-11-03T04:45:54.197Z |  INFO | agency-1 | StageAgencyJob.java | 144 | [19808345:6e2079cf85b23eba] Staging agency finished: JobResult[parsedDepots=Result[value=null, exception=com.vmware.eam.exception.CannotHostUrlException: URL:https://x.x.x.x/bin/vdn/vibs-6.4.10/6.5-17493714/vxlan.zip cant be accessed], vumRes=VumResult[operation=NO_OP, uploadedVibs=Result[value=[], exception=null], baselineRecRes=Result[value=null, exception=null]], pmRes=PmResult[operation=NO_OP, clustersRes=[], depotsRes=Result[value=null, exception=null]]]
2023-11-03T04:45:54.197Z |  INFO | agency-1 | AuditedJob.java | 106 | JOB COMPLETED: [#19808345] StageAgencyJob(Agency ID: bfd62c04-4cd6-4e07-962d-546687f6a09e, Depots: [https://x.x.x.x/bin/vdn/vibs-6.4.10/6.5-17493714/vxlan.zip, https://x.x.x.x/bin/vdn/vibs-6.4.10/6.7-17493714/vxlan.zip], Scope: [ClusterComputeResource:domain-c2307], PM available: false, VUM available: true)

2023-11-03T04:45:54.196Z |  INFO | manage-depots-LRO | Workflow.java | 121 | [LoadDepot:96e7463f47bae98] FAILED
com.vmware.eam.exception.CannotHostUrlException: URL:https://x.x.x.x/bin/vdn/vibs-6.4.10/6.5-17493714/vxlan.zip cant be accessed
 at com.vmware.eam.job.UrlHostJob.call(UrlHostJob.java:107) ~[eam-server.jar:?]
 at com.vmware.eam.job.UrlHostJob.call(UrlHostJob.java:33) ~[eam-server.jar:?]
 at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:58) ~[eam-server.jar:?]
 at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:55) [eam-server.jar:?]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_362]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_362]
 at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
Caused by: com.vmware.eam.security.trust.CannotConnect: Cannot open connection.
 at com.vmware.eam.security.trust.impl.TrustedLeaf.get(TrustedLeaf.java:141) ~[eam-server.jar:?]
 at com.vmware.eam.envoy.route.impl.RouteProvider.provide(RouteProvider.java:193) ~[eam-server.jar:?]
 at com.vmware.eam.envoy.route.impl.RouteProvider.provide(RouteProvider.java:299) ~[eam-server.jar:?]
 at com.vmware.eam.job.UrlHostJob.call(UrlHostJob.java:96) ~[eam-server.jar:?]
 ... 6 more
Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)
 at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_362]

 

 

 

Resolution

1. Remove the EAM baseline. 
2. Remove the EAM agency.

 

- Connect to VCDB and check the table

VPX_EXT_DATA for EXT_ID = 'com.vmware.vim.eam'

Output shall be like the below which shows  baselines which is seen on the vCenter from old NSX V manager.

'ESX Agent Manager :: 56d8c027-f15f-497f-9d89-c6bcb1f13ad1' 

'ESX Agent Manager :: bfd62c04-4cd6-4e07-962d-546687f6a09e'

'ESX Agent Manager :: 82035b01-5c0e-4084-a643-1945cfb34808'

Steps followed: 

 

1. Take snapshot of the vCenter. 

2. Stop the below services. 

>> vmware-vpxd 

>>eam

 

 Post that delete the agent details from the VCDB.

Example:

delete from VPX_EXT_DATA where ext_id = 'com.vmware.vim.eam' and data_key like '%56d8c027-f15f-497f-9d89-c6bcb1f13ad1%';              

 

Restart vCenter services and validate the lifecycle manager to see if the old baseline of EAM is gone. 

Attach the VMware tools baseline and remediate the host.