VKS Cluster Management auto-attach stuck Busy - "cannot update service version 0.1.0 because it is already installed".

Products

• VCF Automation

Issue/Introduction

Following an upgrade of VMware Cloud Foundation Automation (VCFA) from version 9.0.1 to 9.0.2, the VKS Cluster Management service becomes stuck in a "Busy" state within the VCFA UI.

Symptoms observed in the VCFA Provider Portal:

• Navigate to VCF Automation provider portal > Services > Overview > VKS cluster management, the status displays as Busy.
• Navigate to Details > Elements > auto-attach-service shows as "Unhealthy" with the error "Some services are unhealthy : <Region>/<Supervisor>".
• Navigate to Details > Spervisor Services > auto-attach-service shows as "Unhealthy" with error "cannot update service version 0.1.0 because it is already installed".
• This happens each time when VCFA is upgraded from v9.0.1 to v9.0.2.

Environment

VMware Cloud Foundation Automation(VCFA) 9.0.2

vSphere Kubernetes Service (VKS) 9.x

 

Cause

VCFA 9.0.1 installed auto-attach-service version 0.1.0 using an image bundle referencing a specific source SHA. VCFA 9.0.2 ships with the same version string (0.1.0) but references an updated source SHA. vCenter prohibits updating an active supervisor service definition(0.1.0) when the same version is already installed on a supervisor, triggering an upgrade error. Consequently, the ConfigurationHandler element remains in a "Busy" state until the underlying service status tasks are reset.

Resolution

1. Ensure a recent SFTP backup of VCFA is completed prior to performing remediation. Refer On-Demand Backup of VCF Automation. 
2. Uninstall the auto-attach add-on via the VCFA UI:
   
   • Navigate to VCFA Provider portal > Services > Overview > VKS cluster management > Details
   • Click on the three ellipses and select Uninstall.
     
   • Confirm the status changes to Not Installed.
      
3. Prepare to SSH into the VCFA cluster and configure kubeconfig:
   
   ssh vmware-system-user@$vcfa_host
sudo su -
export KUBECONFIG=/etc/kubernetes/admin.conf
   
   
4. Make sure all namespaces(specially "prelude") are active by running command: kubectl get ns
5. Ensure the add-on source is re-downloaded correctly and points to the correct location during the next installation by running the following command on Automation node:
   
   kubectl -n prelude patch addon broadcom.auto-attach-service-0.1.0 --type=json -p='[{"op": "remove", "path": "/status/serviceUnpackageTask"},{"op": "remove", "path": "/status/downloadTask"}]' --subresource=status
   
   *Expected output:*  addon.vcd.addon.vcf.vmware.com/broadcom.auto-attach-service-0.1.0 patched
   
   
6. Reinstall the add-on from the VCFA UI: 

   • Navigate to VCFA Provider portal > Services > Overview > VKS cluster management > Details.

   • Click on the three ellipses and select Install.

   • Follow the prompt instructions and click Submit.
7. Monitor the installation. Within a few minutes, the cluster status will update to Healthy.

Additional Information

If the uninstallation of the auto-attach add-on in VCFA UI becomes stuck, please contact Broadcom Support to resolve the issue.