VKS Guest cluster Ready state shows as False

Products

• VMware vSphere Kubernetes Service
• VMware vCenter Server

Issue/Introduction

• NAPP guest cluster Ready state shows as False.
  

  kubectl get tkc -A -o wide

  NAMESPACE                   NAME                 CONTROL PLANE   WORKER          TKR NAME                          AGE    READY   TKR COMPATIBLE   UPDATES AVAILABLE
<Namespace Name>             <Cluster Name>          3               5        v1.27.6---vmware.1-fips.1-tkg.1      ##d     False   True

• Describing the NAPP cluster from Supervisor Environment shows error related to reconciling.
  

  kubectl describe cluster <cluster-name> -n <namespace>

  error reconciling the Cluster topology: failed to create patch helper for Cluster/<Cluster Name> : server side apply dry-run failed for modified object: admission webhook "tkr-resolver-cluster-webhook.tanzu.vmware.com" denied the request: could not resolve TKR/OSImage for controlPlane, machineDeployments: [worker-nodepool-a1], query: {controlPlane: {k8sVersionPrefix: 'v1.27.6+vmware.1-fips.1', tkrSelector: 'tkr.tanzu.vmware.com/standard', osImageSelector: 'os-name=photon,tkr.tanzu.vmware.com/standard'}, machineDeployments: [{k8sVersionPrefix: 'v1.27.6+vmware.1-fips.1', tkrSelector: 'tkr.tanzu.vmware.com/standard', osImageSelector: 'os-name=photon'}]}, result: {controlPlane: {k8sVersion: '', tkrName: '', osImagesByTKR: map[]}, machineDeployments: [{k8sVersion: '', tkrName: '', osImagesByTKR: map[]}]}
    Reason:                TopologyReconcileFailed
    Severity:              Error
    Status:                False
    Type:                  TopologyReconciled
    Last Transition Time:  YYYY-MM-DD:T:HH:MM:SS
    Reason:                AlreadyUpToDate
    Severity:              Info
    Status:                False
    Type:                  UpdatesAvailable
  Phase:                   unhealthy
  Total Worker Replicas:   5
Events:                    <none>

• In the Supervisor Environment NAPP cluster, new machines have been deployed but are stuck in the provisioning state.
  

  kubectl get machine -o wide -n napp | grep Provisioning  

  NAME                                                       CLUSTER         NODENAME                        PROVIDERID                                       PHASE          AGE     VERSION
machine.cluster.x-k8s.io/<Control plane node>             <Cluster-Name>                                                                                   Provisioning   3m15s   v1.27.6+vmware.1-fips.1
machine.cluster.x-k8s.io/<Worker Node>                    <Cluster-Name>                                                                                   Provisioning   3m15s   v1.27.6+vmware.1-fips.1

Environment

• VMware vSphere Kubernetes Service.

Cause

New nodes are not being deployed because the TKR Resolver webhook failed to find matching TKR/OSImage.

Resolution

To resolve the issue:

• Confirm if the Content Library is accessible by Supervisor Cluster by running the below command.
  kubectl get clustercontentlibraries


• Create a new Subscribed Content Library using Subscription URL. Refer vSphere Supervisor Services and Standalone Components