Supervisor upgrade to VCF 9.1 is stalled with "Solution(s) apply failed on host"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• The Supervisor upgrade in a VMware Cloud Foundation 9.x environment stalls at the host configuration stage.
  
  
• ESXi hosts may remain in an endless "Updating" loop or stay in Maintenance Mode.
  
  
• The vSphere Client reports the following repetitive task:
  Apply Solution,wcp-###########@vsphere.local, A general system error occurred: Solution(s) apply failed on host:##########
  
  
• WCP logs on the vCenter under /var/log/vmware/wcp/wcpsvc.log:
[timestamp] ERROR wcp 2710585 [vc@4413] [pman/client.go:726] [opID=vLCM:Upgrade:domain-c####] PMan API: Task Get for Apply Task API: Attempt#[4]:Failed Attempts:[0] of MaxFailedAttempts[1]Error Message: Solution(s) apply failed on host: '###########'
[timestamp] ERROR wcp 2710585 [vc@4413] [kubelifecycle/pman_client.go:645] [opID=vLCM:Upgrade:domain-c####] PMan API: ApplyUpgradeTask INTERIM FAILURE Error: PMan API: Task Get for Apply Task API: Attempt#[4]:Failed Attempts:[0] of MaxFailedAttempts[1]Error Type: ERROR
[timestamp] ERROR wcp 2710585 [vc@4413] [pman/client.go:682] [opID=vLCM:Upgrade:domain-c####] PMan API: Apply Task: Attempt#[1 of 1]: Has Failed - Task Get Polling failed!
  
  
• Hostd logs on the ESXi host under /var/run/log/hostd.log:
[timestamp] -INFO Hostd 2101486 [esx@4413 sub="Vimsvc.ha-eventmgr"] Event 54755 : Could not install image profile: VMware_bootbank_spherelet_9.0.1.32.5.0-25065159: VMware_bootbank_spherelet_9.0.1.32.5.0-25065159: Failed to unmount tardisk spherele.v00 of VIB VMware_bootbank_spherelet_9.0.1.32.5.0-25065159: Error in running [/bin/rm /tardisks/spherele.v00]:
[timestamp] -INFO Hostd 2101478 [esx@4413] --> Return code: 1
[timestamp] -INFO Hostd 2101478 [esx@4413] --> Output: rm: can't remove '/tardisks/spherele.v00': Device or resource busy

Environment

VMware Cloud Foundation 9.x
VMware vCenter Server
VMware vSphere ESXi

Cause

An incorrect upgrade sequence causes this issue when ESXi hosts upgrade to version 9.1 before the Supervisor upgrade initiates or completes. This sequence results in a version mismatch and an active lock on the Spherelet tardisks, preventing the Supervisor lifecycle manager from updating the host configuration.

Review the VCF 9.0 Update Sequence here: Update Sequence for VCF 9.0 and Compatible VMware Products

Resolution

1. Establish an SSH session to the ESXi host that the wcp service user places into maintenance mode.
2. Stop the Spherelet service manually to release the file lock by running: /etc/init.d/spherelet stop
3. Observe the vSphere Client; the Supervisor host configuration task auto-completes shortly after the Spherelet service stops.
4. Repeat these steps for any additional hosts in the cluster experiencing the same failure.