Automated deployment of the VKS Supervisor VIB is failing
search cancel

Automated deployment of the VKS Supervisor VIB is failing

book

Article ID: 448291

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

  • During the enablement of the VKS Supervisor, the automated deployment of the VKS Supervisor VIB is failing on the target ESXi hosts
  • Manual installation of the VIB via the ESXi CLI is temporarily successful; however, vSphere Lifecycle Manager (vLCM) detects the unauthorized host-level package drift during compliance checks and automatically removes the VIB to enforce the cluster's defined desired-state image.
  • The vCenter file /var/log/vmware/wcp/wcpsvc.log contains the following entries:

<TIMESTAMP> info wcp [pman/client.go:702] [opID=<OP_ID>-host-12622-vLCM:AddHost:domain-<CLUSTER_ID>:host-<HOST_ID>] PMan API: Task Get for Apply Task API: Attempt#[3]:Failed Attempts:[0] of MaxFailedAttempts[1]Starting ...
<TIMESTAMP> info wcp [pman/client.go:713] [opID=<OP_ID>-host-12622-vLCM:AddHost:domain-<CLUSTER_ID>:host-<HOST_ID>] PMan API: Task Get for Apply Task API: Attempt#[3]:Failed Attempts:[0] of MaxFailedAttempts[1]Status = FAILED
<TIMESTAMP> error wcp [pman/client.go:718] [opID=<OP_ID>-host-12622-vLCM:AddHost:domain-<CLUSTER_ID>:host-<HOST_ID>] PMan API: Task Get for Apply Task API: Attempt#[3]:Failed Attempts:[0] of MaxFailedAttempts[1]Error Type: NOT_FOUND
<TIMESTAMP> error wcp [pman/client.go:720] [opID=<OP_ID>-host-12622-vLCM:AddHost:domain-<CLUSTER_ID>:host-<HOST_ID>] PMan API: Task Get for Apply Task API: Attempt#[3]:Failed Attempts:[0] of MaxFailedAttempts[1]Error Message: Solution being applied does not exist in the desired image.
<TIMESTAMP> error wcp [kubelifecycle/pman_client.go:517] [opID=<OP_ID>-host-12622-vLCM:AddHost:domain-<CLUSTER_ID>:host-<HOST_ID>] PMan API: Task Get for Apply Task API: Attempt#[3]:Failed Attempts:[0] of MaxFailedAttempts[1]Error Type: NOT_FOUND
<TIMESTAMP> error wcp [pman/client.go:676] [opID=<OP_ID>-host-12622-vLCM:AddHost:domain-<CLUSTER_ID>:host-<HOST_ID>] PMan API: Apply Task: Attempt#[1 of 1]: Has Failed - Task Get Polling failed!

  • The vCenter file /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log contains the following entries:

<TIMESTAMP> info vmware-vum-server[1477986] [Originator@6876 sub=EntityImageManager] [EntityImageManager 1370] [SoftwareSpecToStr] image spec(JSON String): {
-->     "alternative_images": {},
-->     "base_image": {
-->         "version": "9.0.2.0.25148076"
-->     },
-->     "components": {
-->         "VMware-VM-Tools": "13.1.0-0.25294452"
-->     },
-->     "hardware_support": null,
-->     "removed_components": null,
-->     "solutions": {
-->         "VMware-HBR-Agent": {
-->             "components": [
-->                 {
-->                     "component": "VMware-HBR-Agent"
-->                 }
-->             ],
-->             "version": "9.0.2-0.25093017"
-->         },
-->         "com.vmware.vsphere-wcp": {
-->             "components": [
-->                 {
-->                     "component": "VMware-Spherelet-1-32"
-->                 }
-->             ],
-->             "version": "9.0.1.32.5.0-25065159"
-->         }
-->     }
--> }
...

<TIMESTAMP> error vmware-vum-server[1397061] [Originator@6876 sub=ClusterApplySolutionTask] [ClusterApplySolution, 174] Task:com.vmware.vcIntegrity.lifecycle.ClusterApplySolutionTask ID:5234b86e-a169-3d0c-4e87-d95227f4026e. The solution(solutionId=com.vmware.vsphere-wcp) is not present in the software spec.
<TIMESTAMP> error vmware-vum-server[1397300] [Originator@6876 sub=ClusterApplySolutionTask] [Task, 424] Task:com.vmware.vcIntegrity.lifecycle.ClusterApplySolutionTask ID:5234b86e-a169-3d0c-4e87-d95227f4026e. Task Failed. Error: Error:
-->    com.vmware.vapi.std.errors.not_found
--> Messages:
-->    com.vmware.vcIntegrity.lifecycle.TaskError.SolutionNotFound<Solution being applied does not exist in the desired image.>
--> 
<TIMESTAMP> warning vmware-vum-server[10950] [Originator@6876 sub=TaskStatsCollector] [taskStatsCollector 190] Task type or creation time not present
<TIMESTAMP> info vmware-vum-server[1397300] [Originator@6876 sub=PM.AsyncTask.ClusterApplySolutionTask{125975}] [vciTaskBase 1493] SerializeToVimFault fault:
--> (vmodl.fault.SystemError) {
-->    faultCause = (vmodl.MethodFault) null, 
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.vcIntegrity.lifecycle.TaskError.SolutionNotFound", 
-->          arg = <unset>, 
-->          message = <unset>
-->       }
-->    ], 
-->    reason = "vLCM Task failed, see Error Stack for details."
-->    msg = "{
-->     "data": null,
-->     "error_type": "NOT_FOUND",
-->     "messages": [
-->         {
-->             "args": [],
-->             "default_message": "Solution being applied does not exist in the desired image.",
-->             "id": "com.vmware.vcIntegrity.lifecycle.TaskError.SolutionNotFound"
-->         }
-->     ]
--> }"

  • The vCenter web UI repeatedly shows the following failing task:

The Workload Control Plane (wcp-<guid>) initiates repetitive cluster remediation tasks that fail with: "A general system error occurred: Solution being applied does not exist in the desired image".

Environment

vCenter 9.0.2

Resolution

1) From the vCenter server, retrieve all the WCP offline bundles by running: 

dcli com vmware esx settings depots offline list | grep -B3 -A3 'WCP offline bundle'

         Example output:

60654AE09465E4017B6908A48B4C699AF0000A6B:
   owner: com.vmware.vcIntegrity
   file_id: /storage/updatemgr/patch-store-temp/vsphere-wcp-depot.zip
   description: WCP offline bundle
   ownerdata: com.vmware.vsphere-wcp_8.0.3-25092719
   location:
   source_type: PUSH
7CC592BE80A88C2E2B8BE43F2B76C8450D640D8C:
   owner: com.vmware.vcIntegrity
   file_id: /storage/updatemgr/patch-store-temp/vsphere-wcp-depot.zip
   description: WCP offline bundle
   ownerdata: com.vmware.vsphere-wcp_8.0.3-24853646
   location:
   source_type: PUSH
A89802D8E81FD3F7427A9D395A9B15DD7D3BAF6D:
   owner: com.vmware.vcIntegrity
   file_id: /storage/updatemgr/patch-store-temp/vsphere-wcp-depot-embedded.zip
   description: WCP offline bundle
   ownerdata: com.vmware.vsphere-wcp
   location:
   source_type: PUSH
FBFB65D38D7C4286E92E02634E87043C82761927:
   owner: com.vmware.vcIntegrity
   file_id: /storage/updatemgr/patch-store-temp/vsphere-wcp-depot-embedded.zip
   description: WCP offline bundle
   ownerdata: com.vmware.vsphere-wcp
   location:
   source_type: PUSH        

2) Attempt to delete all WCP offline bundles in the depot (You'll have a WCP offline bundle for each time your vCenter has been upgraded). For example:

dcli com vmware esx settings depots offline delete-task --depot 60654AE09465E4017B6908A48B4C699AF0000A6B
dcli com vmware esx settings depots offline delete-task --depot 7CC592BE80A88C2E2B8BE43F2B76C8450D640D8C
dcli com vmware esx settings depots offline delete-task --depot A89802D8E81FD3F7427A9D395A9B15DD7D3BAF6D
dcli com vmware esx settings depots offline delete-task --depot FBFB65D38D7C4286E92E02634E87043C82761927

         NOTE: You can ignore the following error messages:

Server error: com.vmware.vapi.std.errors.InvalidArgument
Error message:
    The system-defined offline depot '60654AE09465E4017B6908A48B4C699AF0000A6B' contains the latest WCP solution version 8.0.3-25092719, which is NOT allowed to be deleted. Otherwise, the corresponding feature won't be enabled on clusters properly.

3) Restart the VMware Update Manager (VUM) service:

vmon-cli -r updatemgr

4) Wait for the Set Solution task to finish in vCenter

5) Verify that you have a single WCP offline bundle in the depot list:

dcli com vmware esx settings depots offline list | grep -B3 -A3 'WCP offline bundle'

         Example output:

FBFB65D38D7C4286E92E02634E87043C82761927:
   owner: com.vmware.vcIntegrity
   file_id: /storage/updatemgr/patch-store-temp/vsphere-wcp-depot-embedded.zip
   description: WCP offline bundle
   ownerdata: com.vmware.vsphere-wcp
   location:
   source_type: PUSH

6) Verify that the VUM server is able to find and load the WCP solution:

grep "Preload" /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

<TIMESTSAMP> info vmware-vum-server[26735] [Originator@6876 sub=VumVapiEndpoint] [plugin 3406] Preload system-defined offline bundles.
<TIMESTSAMP> info vmware-vum-server[26735] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask] [Task, 321] Task:com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask ID:52f31401-c99c-7ad3-ba1c-5d7a03707bd5. Task Created
<TIMESTSAMP> info vmware-vum-server[26633] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask] [PreloadOfflineBundlesTask 135] Copying bundle[ESXi-Minimal-Depot] offline bundle from: /usr/lib/vmware-updatemgr/minimal-lts-depot/esxi-minimal-depot.zip to: /storage/updatemgr/patch-store-temp/esxi-minimal-depot.zip
<TIMESTSAMP> info vmware-vum-server[26633] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask] [Task 631] Set com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask (52f31401-c99c-7ad3-ba1c-5d7a03707bd5) progress to 6
<TIMESTSAMP> info vmware-vum-server[26633] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask] [Task 631] Set com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask (52f31401-c99c-7ad3-ba1c-5d7a03707bd5) progress to 33
<TIMESTSAMP> info vmware-vum-server[26633] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask] [PreloadOfflineBundlesTask 135] Copying bundle[HA] offline bundle from: /etc/vmware-vpx/docRoot/vSphere-HA-depot/vsphere-ha-depot.zip to: /storage/updatemgr/patch-store-temp/vsphere-ha-depot.zip
<TIMESTSAMP> info vmware-vum-server[26633] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask] [Task 631] Set com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask (52f31401-c99c-7ad3-ba1c-5d7a03707bd5) progress to 39
<TIMESTSAMP> info vmware-vum-server[26633] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask] [Task 631] Set com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask (52f31401-c99c-7ad3-ba1c-5d7a03707bd5) progress to 66
<TIMESTSAMP> info vmware-vum-server[26633] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask] [PreloadOfflineBundlesTask 135] Copying bundle[WCP] offline bundle from: /storage/lifecycle/vmware-wcp/spherelet/vsphere-wcp-depot-embedded.zip to: /storage/updatemgr/patch-store-temp/vsphere-wcp-depot-embedded.zip
<TIMESTSAMP> info vmware-vum-server[26633] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask] [Task 631] Set com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask (52f31401-c99c-7ad3-ba1c-5d7a03707bd5) progress to 72
<TIMESTSAMP> info vmware-vum-server[26633] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask] [Task 631] Set com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask (52f31401-c99c-7ad3-ba1c-5d7a03707bd5) progress to 99
<TIMESTSAMP> info vmware-vum-server[26633] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask] [Task, 505] Task:com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask ID:52f31401-c99c-7ad3-ba1c-5d7a03707bd5. Finalizing Task
<TIMESTSAMP> info vmware-vum-server[26633] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask] [Task, 345] Task:com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask ID:52f31401-c99c-7ad3-ba1c-5d7a03707bd5. Task Finalization completed.
<TIMESTSAMP> info vmware-vum-server[26633] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask] [Task, 464] Task:com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask ID:52f31401-c99c-7ad3-ba1c-5d7a03707bd5. Task Execution completed
<TIMESTSAMP> info vmware-vum-server[26633] [Originator@6876 sub=PM.AsyncTask.PreloadOfflineBundlesTask{1}] [vciTaskBase 1119] Task finished, state: 2
<TIMESTSAMP> info vmware-vum-server[26633] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask] [Task, 470] Task:com.vmware.vcIntegrity.lifecycle.PreloadOfflineBundlesTask ID:52f31401-c99c-7ad3-ba1c-5d7a03707bd5. Task State updated to SUCCEEDED
<TIMESTSAMP> info vmware-vum-server[26738] [Originator@6876 sub=Telemetry] [TelemetryManager 265] Sending telemetry data: {"@type":"pman_task","taskId":"2bab3e7a-620c-4992-86e8-3d3f15043101|52f31401-c99c-7ad3-ba1c-5d7a03707bd5","opId":"","succeeded":true,"entityId":"2bab3e7a-620c-4992-86e8-3d3f15043101|","taskName":"PreloadOfflineBundlesTask","startTime":"2026-07-01T18:51:49.84227Z","endTime":"2026-07-01T18:52:22.184806Z"}

7) Verify the WCP VIB was installed on the ESXi hosts by establishing an ssh connection to each ESXi host in the cluster and running:

esxcli software vib list | grep spherelet
spherelet                      9.0.1.32.5.0-25065159                 VMware  VMwareCertified   2026-06-12    host

8) Verify the vspherelet service is running:

/etc/init.d/spherelet status
2026-07-20 17:32:36,768 init.d/spherelet spherelet init script invoked via the following hierarchy
2026-07-20 17:32:36,768 init.d/spherelet 3459400: -sh
2026-07-20 17:32:36,768 init.d/spherelet 3459398: /bin/sshd-session -i -R
2026-07-20 17:32:36,768 init.d/spherelet 3459392: /bin/sshd-session -i -R
2026-07-20 17:32:36,768 init.d/spherelet 1049327: /bin/inetd /var/run/inetd.conf -f
2026-07-20 17:32:36,768 init.d/spherelet 1048787: /bin/init
2026-07-20 17:32:36,914 init.d/spherelet Log fetcher support: True
2026-07-20 17:32:36,914 init.d/spherelet Log fetcher size: 460
2026-07-20 17:32:36,927 init.d/spherelet spherelet is running
2026-07-20 17:32:36,927 init.d/spherelet spherelet is running

 

Additional Information

Using DCLI with a Credential Store File

Powered by Wolken Software