1. Unable to install VMtools or ESXi patch via lifecycle manager.
2. No issues found while patching via command line.
3. Update initiated by new baseline with "automatically updates" unchecked also fails.
4.Error received on the UI :An error occured during host configuration
/usr/sbin/esxupdate returned with exit status :32
Infra has NSX installed.
Log location: /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log
The patching failing with error in logs stating its failing to install the nsx vibs.
2023-11-03T14:57:59.488+08:00 info vmware-vum-server[177403] [Originator@6876 sub=HostUpdateDepotManager] [vibDownloaderImpl 361] Downloaded and validated vibs successfully
2023-11-03T14:57:59.488+08:00 info vmware-vum-server[177403] [Originator@6876 sub=HostUpdateDepotManager] [patchDepotManager 692] Vib file: /storage/updatemgr/patch-store/hostupdate/vmw/vib20/esx-nsxv/VMware_bootbank_esx-nsxv_6.7.0-0.0.17493714.vibis already downloaded (and being copied).
2023-12-20T11:03:25.637+08:00 info vmware-vum-server[422112] [Originator@6876 sub=HostUpdateDepotManager] [scanHostxxx] List of updates for installation/staging on host: xxx (entity: host-xxx):
--> -----------------------------------------------
--> (VMware_bootbank_esx-nsxv_6.7.0-0.0.17493714, VMware_locker_tools-light_12.3.5.22544099-22545121)
--> ------------------------------------------------
2023-12-20T11:03:25.637+08:00 info vmware-vum-server[422112] [Originator@6876 sub=SingleHostRemediateTask.SingleHostRemediateTask{375}] [singleHostRemediateTaskBase 392]
--> ------------------------------------------------
--> (VMware_bootbank_esx-nsxv_6.7.0-0.0.17493714, VMware_locker_tools-light_12.3.5.22544099-22545121)
--> ------------------------------------------------
The baseline VM Tools doesn't contain nsxv
2023-10-27T12:24:14.282+08:00 info vmware-vum-server[09460] [Originator@6876 sub=BaselineMgr] [baselineMgr 5497] Baseline 'VMWare Tools
12.3.0' with ID '21' has 'TOOLS-22236987', updates
However the EAM baselines does seem to contain nsxv vib in the updates list
2023-10-27T12:24:14.308+08:00 info vmware-vum-server[09460] [Originator@6876 sub=BaselineMgr] [baselineMgr 5497] Baseline 'ESX Agent Manager
:: 56d8c027-f15f-497f-9d89-c6bcb1f13ad1' with ID '10' has 'nsxv-datapath-6.7-17493714', 'nsxv-datapath-6.5-17493714', updates.
2023-10-27T12:24:14.350+08:00 info vmware-vum-server[09460] [Originator@6876 sub=BaselineMgr] [baselineMgr 5497] Baseline 'ESX Agent Manager
:: bfd62c04-4cd6-4e07-962d-546687f6a09e' with ID '8' has 'nsxv-datapath-6.7-17493714', 'nsxv-datapath-6.5-17493714', updates.
2023-10-27T12:24:14.385+08:00 info vmware-vum-server[09460] [Originator@6876 sub=BaselineMgr] [baselineMgr 5497] Baseline 'ESX Agent Manager
:: 82035b01-5c0e-4084-a643-1945cfb34808' with ID '9' has 'nsxv-datapath-6.7-17493714', 'nsxv-datapath-6.5-17493714', updates.
Cause:
EAM may stage and install VIBs through VUM depending on the VIB type
There are legacy VIBs that are installed through VUM and there are vLCM VIBs that are installed through vLCM
Eam.log
2023-11-03T04:45:29.547Z | INFO | vim-monitor | ExecutorUtils.java | 36 | JOB SUBMITED: [#19808345] StageAgencyJob(Agency ID: bfd62c04-4cd6-4e07-962d-546687f6a09e, Depots: [https://x.x.x.x/bin/vdn/vibs-6.4.10/6.5-17493714/vxlan.zip, https://x.x.x.x/bin/vdn/vibs-6.4.10/6.7-17493714/vxlan.zip], Scope: [ClusterComputeResource:domain-c2307], PM available: false, VUM available: true)
2023-11-03T04:45:29.576Z | INFO | agency-1 | AuditedJob.java | 106 | JOB STARTED: [#19808345] StageAgencyJob(Agency ID: bfd62c04-4cd6-4e07-962d-546687f6a09e, Depots: [https://x.x.x.x/bin/vdn/vibs-6.4.10/6.5-17493714/vxlan.zip, https://x.x.x.x/bin/vdn/vibs-6.4.10/6.7-17493714/vxlan.zip], Scope: [ClusterComputeResource:domain-c2307], PM available: false, VUM available: true)
2023-11-03T04:45:47.950Z | ERROR | host-292524-0 | VibJob.java | 300 | Cannot access vib package at https://x.x.x.x/bin/vdn/vibs-6.4.10/6.7-17493714/vxlan.zip
com.vmware.eam.security.trust.CannotConnect: Cannot open connection.
2023-11-03T04:45:54.197Z | INFO | agency-1 | StageAgencyJob.java | 144 | [19808345:6e2079cf85b23eba] Staging agency finished: JobResult[parsedDepots=Result[value=null, exception=com.vmware.eam.exception.CannotHostUrlException: URL:https://x.x.x.x/bin/vdn/vibs-6.4.10/6.5-17493714/vxlan.zip cant be accessed], vumRes=VumResult[operation=NO_OP, uploadedVibs=Result[value=[], exception=null], baselineRecRes=Result[value=null, exception=null]], pmRes=PmResult[operation=NO_OP, clustersRes=[], depotsRes=Result[value=null, exception=null]]]
2023-11-03T04:45:54.197Z | INFO | agency-1 | AuditedJob.java | 106 | JOB COMPLETED: [#19808345] StageAgencyJob(Agency ID: bfd62c04-4cd6-4e07-962d-546687f6a09e, Depots: [https://x.x.x.x/bin/vdn/vibs-6.4.10/6.5-17493714/vxlan.zip, https://x.x.x.x/bin/vdn/vibs-6.4.10/6.7-17493714/vxlan.zip], Scope: [ClusterComputeResource:domain-c2307], PM available: false, VUM available: true)
2023-11-03T04:45:54.196Z | INFO | manage-depots-LRO | Workflow.java | 121 | [LoadDepot:96e7463f47bae98] FAILED
com.vmware.eam.exception.CannotHostUrlException: URL:https://x.x.x.x/bin/vdn/vibs-6.4.10/6.5-17493714/vxlan.zip cant be accessed
at com.vmware.eam.job.UrlHostJob.call(UrlHostJob.java:107) ~[eam-server.jar:?]
at com.vmware.eam.job.UrlHostJob.call(UrlHostJob.java:33) ~[eam-server.jar:?]
at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:58) ~[eam-server.jar:?]
at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:55) [eam-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
Caused by: com.vmware.eam.security.trust.CannotConnect: Cannot open connection.
at com.vmware.eam.security.trust.impl.TrustedLeaf.get(TrustedLeaf.java:141) ~[eam-server.jar:?]
at com.vmware.eam.envoy.route.impl.RouteProvider.provide(RouteProvider.java:193) ~[eam-server.jar:?]
at com.vmware.eam.envoy.route.impl.RouteProvider.provide(RouteProvider.java:299) ~[eam-server.jar:?]
at com.vmware.eam.job.UrlHostJob.call(UrlHostJob.java:96) ~[eam-server.jar:?]
... 6 more
Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_362]
1. Remove the EAM baseline.
2. Remove the EAM agency.
- Connect to VCDB and check the table
VPX_EXT_DATA for EXT_ID = 'com.vmware.vim.eam'
Output shall be like the below which shows baselines which is seen on the vCenter from old NSX V manager.
'ESX Agent Manager :: 56d8c027-f15f-497f-9d89-c6bcb1f13ad1'
'ESX Agent Manager :: bfd62c04-4cd6-4e07-962d-546687f6a09e'
'ESX Agent Manager :: 82035b01-5c0e-4084-a643-1945cfb34808'
Steps followed:
1. Take snapshot of the vCenter.
2. Stop the below services.
>> vmware-vpxd
>>eam
Post that delete the agent details from the VCDB.
Example:
delete from VPX_EXT_DATA where ext_id = 'com.vmware.vim.eam' and data_key like '%56d8c027-f15f-497f-9d89-c6bcb1f13ad1%';
Restart vCenter services and validate the lifecycle manager to see if the old baseline of EAM is gone.
Attach the VMware tools baseline and remediate the host.