kubernetes guest cluster creation failing with "Multiple OSImages resolved for Control Plane"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Scenarios:

• You attempt to create (or upgrade) a cluster.
  
  
• You receive the following error:

admission webhook "tkr-resolver-cluster-webhook.tanzu.vmware.com" denied the request: Could not resolve KR/OSImage Multiple OSImages resolved for Control Plane.

• Example, when creating a cluster using TKR v1.30.1 you see:

Error from server (Forbidden): error when creating "cluster.yaml": admission webhook "tkr-resolver-cluster-webhook.tanzu.vmware.com" denied the request: Could not resolve KR/OSImage
Multiple OSImages resolved for Control Plane. KR: v1.30.1---vmware.1-fips-tkg.5, OSImages resolved: [vmi-################ vmi-################], filters: {k8sVersionPrefix: v1.30.1+vmware.1-fips, osImageSelector: os-name=ubuntu,tkr.tanzu.vmware.com/standard}
Multiple OSImages resolved for MachineDeployment default. KR: v1.30.1---vmware.1-fips-tkg.5, OSImages resolved: [vmi-################ vmi-################], filters: {k8sVersionPrefix: v1.30.1+vmware.1-fips, osImageSelector: os-name=ubuntu}

Environment

VMware Tanzu Kubernetes Service

vSphere Supervisor: Kubernetes version v1.27

vCenter 8.0.3e

Cause

You may have multiple Content Libraries linked to the Supervisor environment and they contain the same VKr images

Refer to Resolve VKr Image Conflicts.

Or you may be able to dissociate or remove one of the libraries completely (see Resolution section)

Resolution

Confirm if multiple Content Libraries are linked to the Supervisor environment

• Check the Supervisor using kubectl CLI

Using Supervisor kubectl context, run the following command:

kubectl get clustercontentlibraries

Example:

$ kubectl get clustercontentlibraries 
NAME                     VSPHERENAME       TYPE          STORAGETYPE   AGE
cl-CONTENT_LIBRARY_ID1   LIB_NAME_1        Subscribed    Datastore     157d
cl-CONTENT_LIBRARY_ID2   LIB_NAME_2        Subscribed    Datastore     157d

 

• If you see more than one Content Library, determine if you can remove a 2nd Content Library entirely
  
  
  • Note the Names of each Content Library from above. 
    
    
  • Confirm if Content Libraries contain the same items using clustercontentlibraryitems.  Use the following commands to confirm the same library items.

kubectl get clustercontentlibraryitems --sort-by='{.status.contentLibraryRef.name}' --no-headers  | awk '{print $1,$2,$3}' | sort -n | grep cl-CONTENT_LIBRARY_ID1

kubectl get clustercontentlibraryitems --sort-by='{.status.contentLibraryRef.name}' --no-headers  | awk '{print $1,$2,$3}' | sort -n | grep cl-CONTENT_LIBRARY_ID12 

 

• If the content library items are the same
  
  
  • Disassociate one of the Content Libraries from the vSphere -> Workload Management -> Namespace -> "VM Service" Tile.  Remove the duplicate from "Associated Content Libraries" if it exists.
    
    Example showing 0 from that tile:
    
    
    
    
    OR
    
    
    
  • Remove one of the Content Libraries entirely from vSphere:   via vSphere UI -> Content Libraries
    
    

 

If the content libraries are 'not' the same, or you cannot remove one one of them, you will need to add the documented annotation to your cluster.

Refer to Resolve VKr Image Conflicts.