Deployment of Ubuntu OS-Based Nodes on Guest Cluster Fails with “Missing Compatible KR/OSImage for the Cluster” Error

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• When attempting to deploy a Guest Cluster using Ubuntu OS, the deployment fails with the following error message : 
  

  Error from server (Spec.Topology.ControlPlane.TKR.Reference.Name unable to resolve that Kubernetes release due to Could not resolve KR/OSImage
  Missing compatible KR/OSImage for the cluster
  Control Plane, filters: {k8sVersionPrefix: '<TKR.VERSION>', osImageSelector: os-name='<OS NAME>'/standard}
  MachineDeployment worker-nodepool, filters: {k8sVersionPrefix: '<TKR.VERSION>', osImageSelector: os-name='<OS NAME>'}
  ): error when creating ".\\'<CLUSTER_NAME>'": admission webhook "default.validating.tanzukubernetescluster.run.tanzu.vmware.com" denied the request: Spec.Topology.ControlPlane.TKR.Reference.Name unable to resolve that Kubernetes release due to Could not resolve KR/OSImage
  Missing compatible KR/OSImage for the cluster

  
  
  
• The Guest Cluster deployment fails, and control plane or worker nodes based on Ubuntu OS are not created.

Environment

vSphere Kubernetes Service

Cause

The issue occurs because the Ubuntu OS annotation was missing from the cluster deployment YAML. By default, vSphere Kubernetes Clusters use Photon OS images unless explicitly directed otherwise. When the Ubuntu annotation is not included, the system attempts to resolve a Photon OS image for the specified Kubernetes release (TKR).

If your environment is using a local content library with only ubuntu images, the above system defaults will search for a Photon image that does not exist.

Resolution

To successfully deploy Ubuntu-based nodes, the following steps must be performed: 

1. Verify Ubuntu OS image availability : 
   
   Run the following command to ensure that an Ubuntu OS image exists for the required Tanzu Kubernetes Release (TKR):
   

   kubectl get osimages -n <namespace_name> | grep -i ubuntu

    

2. Check the affected workload cluster's annotations for control planes and all node-pools for worker nodes:
   

   kubectl describe cluster -n <namespace> <affected workload cluster name> | grep -i annotations -A2 -B6
   
       controlPlane:
         replicas: #
         metadata:
           annotations:
             run.tanzu.vmware.com/resolve-os-image: os-name=<OS>
   
   workers:
         machineDeployments:
           - class: node-pool
             name: <node-pool-name>
             replicas: #
             metadata:
               annotations:
                 run.tanzu.vmware.com/resolve-os-image: os-name=<OS>

3. If missing, add Ubuntu OS annotation in the cluster YAML :
   
   Edit the cluster YAML and add the following annotation under the metadata section:
   

   metadata:
     name: tkc-ubuntu-gpu
     namespace: tkg-cluster-ns
     annotations:
       run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu

   This annotation instructs the system to use an Ubuntu OS image instead of Photon OS during cluster provisioning. 
   
   NOTE : The configuration syntax depends on the Cluster API version being used by the Guest Cluster. For example : 
                1. Clusters with API version v1alpha3, include the annotation as mentioned in the following document - v1alpha3 Example: TKC with Ubuntu TKR
                2. Clusters with API version v1beta1, include the annotation as mentioned in the following document - v1beta1 Example: Cluster with Ubuntu TKR
    
4. Verify from VKR Release Notes if the desired VKR version requires a specific Ubuntu version.
   Edit the cluster YAML annotation to include the desired Ubuntu OS version:
   

   run.tanzu.vmware.com/resolve-os-image: "os-name=ubuntu,os-version=<OS version>"

    

5. Once the annotation is completed, reapply the YAML for cluster deployment. 
   

   kubectl apply -f <cluster-yaml-file>