vSphere Supervisor Content Library Images and KRs Missing After Content Library Detach and Reattach to the Supervisor Cluster

Products

• Tanzu Kubernetes Runtime
• VMware vSphere Kubernetes Service

Issue/Introduction

If a content library is detached then re-attached to the Supervisor cluster immediately, the corresponding content library resources will not show up on the Supervisor cluster afterwards.

While connected to the Supervisor cluster context, one or more of the following symptoms are observed:

• There are fewer than expected or no resources found for the Kubernetes Releases (KR) output from the below command:
  

  kubectl get kr

• The content library associated with the Supervisor cluster is not found, despite being attached in the vSphere web UI:
  

  kubectl get clustercontentlibrary
  
  No resources found.

• Workload Clusters cannot be created or scaled due to the missing Kubernetes Releases.

Environment

vSphere 9.0

VKS service v3.0.0 and higher

Cause

When a content library is detached from a Supervisor cluster, the corresponding Kubernetes resources will be cleaned up by the system.

However in this scenario, the content library has been re-attached to the Supervisor cluster before the Kubernetes resources have finished deleting.

This timing prevents the corresponding Kubernetes resources from being recreated after the content library re-attachment.

VM service content libraries are affected in the same manner when detached then re-attached before its corresponding cluster virtual machine image resources are cleaned up.

Resolution

Resolution

This will be fixed in vSphere 9.0.1.

 

Workaround

Confirm that the corresponding Kubernetes resources were cleaned up successfully by the system then re-attach the content library to the Supervisor cluster again.

IMPORTANT: Manual deletions of the resources is not necessary because the system should quickly clean up the resources.

1. Locate the detached content library's ID from the vSphere web UI, the below image's content library ID is replaced with #'s:
   
   
   
2. Connect into the Supervisor cluster context
   
   
3. List the cluster content libraries associated with the Supervisor cluster:
   

   kubectl get clustercontentlibrary
   
   NAME                                                                  VSPHERENAME                          TYPE         STORAGETYPE
   clustercontentlibrary.imageregistry.vmware.com/cl-<contentlibraryid>   Kubernetes Service Content Library  Subscribed   Datastore

    

4. Describe the clustercontentlibrary to find the one that uses the same UUID as the detached content library from Step 1:
   

   kubectl describe clustercontentlibrary <clustercontentlibrary item> | grep -i uuid

    

5. Check for the following kubernetes resources that reference the clustercontentlibrary's <contentlibraryid> from Step 3 and 4:
   

   kubectl get clustercontentlibrary,clustercontentlibraryitem | grep <contentlibraryid from Step 3 and 4>
   
   clustercontentlibrary.imageregistry.vmware.com/cl-<contentlibraryid>   Kubernetes Service Content Library  Subscribed   Datastore
   clustercontentlibraryitem.imageregistry.vmware.com/clitem-<item id>  <ovf image name>                 cl-<contentlibraryid>       OVF

    

6. Confirm that the below kubernetes resources are cleaned up successfully:
   

   kubectl get cvmi,kr

    

7. Once the above kubernetes resources from Step 4 and 5 (which were associated with the removed content library) are deleted successfully, the content library can be re-attached to the Supervisor cluster.