Upgrade of Workload Cluster Fails If Status of Target Tanzu Kubernetes Release is unknown

Products

• Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:

• When updating a Tanzu Kubernetes Grid (TKG) / Telco Cloud Automation (TCA) workload cluster you see messages similar to the following from tkg or tanzu CLI:

Failed to upgrade the cluster cdc-work-cluster-v1-20-8. Reason: compatibility file (/root/.config/tanzu/tkg/compatibility/tkg-compatibility.yaml) already exists, skipping download
BOM files inside /root/.config/tanzu/tkg/bom already exists, skipping download
Error: no available upgrades for cluster "cdc-work-cluster-v1-20-8", namespace "cdc-work-cluster-v1-20-8"

• You see that one or more Tanzu Kubernetes Release (tkr) objects with a COMPATIBLE field showing as UNKNOWN:

kubectl get tkr

NAME                              VERSION                         COMPATIBLE   CREATED
v1.20.4---vmware.1-tkg.2          v1.20.4+vmware.1-tkg.2          False        12h
v1.20.4---vmware.3-tkg.1          v1.20.4+vmware.3-tkg.1          False        12h
v1.20.5---vmware.1-tkg.1          v1.20.5+vmware.1-tkg.1          False        12h
v1.20.5---vmware.2-fips.1-tkg.1   v1.20.5+vmware.2-fips.1-tkg.1   False        12h
v1.20.5---vmware.2-tkg.1          v1.20.5+vmware.2-tkg.1          False        12h
v1.20.8---vmware.1-tkg.2          v1.20.8+vmware.1-tkg.2          Unknown      12h
v1.21.2---vmware.1-tkg.1          v1.21.2+vmware.1-tkg.1          True         12h

kubectl get tkr v1.20.8---vmware.1-tkg.2 -o yaml

apiVersion: run.tanzu.vmware.com/v1alpha1
kind: TanzuKubernetesRelease
metadata:
  creationTimestamp: "2021-09-26T18:25:37Z"
  generation: 1
  name: v1.20.8---vmware.1-tkg.2
  resourceVersion: "2929"
  uid: ########-####-####-####-############
spec:
  images:
  .....
    repository: projects.registry.vmware.com/tkg
    tag: v3.2.0_vmware.1
  - name: csi/csi-provisioner
    repository: projects.registry.vmware.com/tkg
    tag: v2.2.0_vmware.1
  kubernetesVersion: v1.20.8+vmware.1
  repository: projects.registry.vmware.com/tkg
  version: v1.20.8+vmware.1-tkg.2
status:
  conditions:
  - lastTransitionTime: "2021-09-26T18:25:36Z"
    status: Unknown
    type: Compatible
  - lastTransitionTime: "2021-09-26T18:25:36Z"
    status: "False"
    type: UpdatesAvailable
  - lastTransitionTime: "2021-09-26T18:25:36Z"
    message: Deprecated
    status: "False"
    type: UpgradeAvailable

Environment

1.4

Cause

• It is expected to not be able to apply a tkr whose status is UNKNOWN

• The tkr status is set to unknown because when validating tkr compatibility the discovery frequency on tkr-controller-manage has timed out

Resolution

Upgrade to TKG 1.4.0+

Workaround:

To workaround this issue, delete the tkr-controller pod(s) in the tkr-system namespace using kubectl so they are automatically recreated and retry the compatibility discovery.

Additional Information