TKR showing 'False' after TKG service upgrade to compatible version

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• TKG service has been upgraded to 3.1.1 and as per Product Interoperability Matrix, the TKR 1.30.1 should be compatible with 3.1.1 version of TKG service but shows 'False' in compatibility list. The latest compatible image according to kubectl get tkr shows TKR 1.29.4.
  
  
• PKGI for the TKG-service shows reconcile succeeded.
  
  kubectl get pkgi -A | grep -i svc-tkg
vmware-system-supervisor-services   svc-tkg.vsphere.vmware.com     tkg.vsphere.vmware.com        3.1.1        Reconcile succeeded   143d
  
  
• Both cvmi and osimage are available on the Supervisor for the 'False' tkr:
  
kubectl get osimage,cvmi | grep v1.30.1
osimage.run.tanzu.vmware.com/vmi-1cb39fb35de19b4f1   v1.30.1+vmware.1-fips      photon    5            amd64   vmi                 6m53s
osimage.run.tanzu.vmware.com/vmi-4c4257440235588ed   v1.30.1+vmware.1-fips      ubuntu    22.04        amd64   vmi                 130d
clustervirtualmachineimage.vmoperator.vmware.com/vmi-1cb39fb35de19b4f1   ob-24075942-photon-5-amd64-vmi-k8s-v1.30.1---vmware.1-fips-tkg.5            v1.30.1+vmware.1-fips-tkg.5             vmwarePhoton64Guest                17
clustervirtualmachineimage.vmoperator.vmware.com/vmi-4c4257440235588ed   ob-24076161-ubuntu-2204-amd64-vmi-k8s-v1.30.1---vmware.1-fips-tkg.5         v1.30.1+vmware.1-fips-tkg.5             ubuntu64Guest                      17
  
  
• Restarting the tkg-controller and tkg-resolver pods on the Supervisor does not resolve the issue—the behavior remains unchanged.
  
  
• The compat file still has TKGSvS as "3.0.0".
  
kubectl_get_yaml_compat.txt
"TKGSvS": {
            "versions": {
              "vmware.com/gccontroller": [
                "3.0.0"
              ]
            }
          },


• Even after upgrading the TKG service to a more higher version (3.2.0) gives the TKGSvS version as 3.0.0.
  
  
• tkg-controller-manager logs on Supervisor shows below entries:
  
  yyyy-mm-dd check.go:408] "msg"="Requirement not met in compatibility document" "logger"="compatibility" "predicate"={"Operation":"isVersionSatisfied","ArgumentsForLogicalOperators":null,"ArgumentsForFunctions":{"initiator":"vmware.com/gcauthservice","receiver":"vmware.com/gccontroller","versions":["3.1.0"]}} "requirement"="TKGSvS"

Environment

VMware Kubernetes Service
vSphere with Tanzu

Cause

Although the TKG service indicates a successful upgrade, the underlying applications remain on earlier versions. As a result, the TKRs are marked as incompatible and appear as 'False'.

Resolution

• Review the package install resources to find the status of "paused:true":
  
kubectl get pkgi pkgi svc-tkg.vsphere.vmware.com -n vmware-system-supervisor-services -o yaml
kubectl get pkgi pkgi tanzu-cluster-api -n <namespace> -o yaml


• Upause the below packageinstalls if found in paused:true state.
  
  
• • name: svc-tkg.vsphere.vmware.com
    kubectl patch pkgi svc-tkg.vsphere.vmware.com -n vmware-system-supervisor-services --type merge -p '{"spec":{"paused":false}}'
    
    
  • name: tanzu-cluster-api
    kubectl patch pkgi tanzu-cluster-api -n <namespace> --type merge -p '{"spec":{"paused":false}}'