TKG 2.1 upgrade to TKG 2.2 fails when searching for 1.25.7 Tanzu Kubernetes Release on Air Gapped environment

Products

• Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:

• When upgrading TKGm 2.1 to TKG 2.2 the procedure fails with an error:
  
  could not resolve TKR/OSImage for controlPlane, machineDeployments: [md-0 md-1 md-2], query: {controlPlane: {k8sVersionPrefix: 'v1.25.7+vmware.2-tkg.1', tkrSelector: '', osImageSelector: 'image-type=ova,os-name=ubuntu'}, machineDeployments: [{k8sVersionPrefix: 'v1.25.7+vmware.2-tkg.1', tkrSelector: '', osImageSelector: 'image-type=ova,os-name=ubuntu'} {k8sVersionPrefix: 'v1.25.7+vmware.2-tkg.1', tkrSelector: '', osImageSelector: 'image-type=ova,os-name=ubuntu'} {k8sVersionPrefix: 'v1.25.7+vmware.2-tkg.1', tkrSelector: '', osImageSelector: 'image-type=ova,os-name=ubuntu'}]}, result: {controlPlane: {k8sVersion: '', tkrName: '', osImagesByTKR: map[]}, machineDeployments: [{k8sVersion: '', tkrName: '', osImagesByTKR: map[]} {k8sVersion: '', tkrName: '', osImagesByTKR: map[]} {k8sVersion: '', tkrName: '', osImagesByTKR: map[]}]}", "UID": "4b7051a0-f655-45c9-b1f1-336bbae5da54", "allowed": false}
  
  
• Running kubectl get tkr does not list the new TKRs for TKGm 2.2, and the upgrade cannot proceed.
  
  

Environment

VMware Tanzu Kubernetes Grid 2.1.1
VMware Tanzu Kubernetes Grid 2.4.0
VMware Tanzu Kubernetes Grid 2.3.0

Cause

The tkr-source-controller do not find the tkr-vsphere-nonparavirt-v1.25.7_vmware.2-tkg.1 and tkg-vsphere-repository-nonparavir-1.25.7_vmware.2-tkg.1 package on Harbor

Resolution

Upload the missing packages manually using imgpkg to Harbor.

Workaround:

Follow these steps when uploading the bundle to the private Harbor (as stated in the  Prepare an Internet-Restricted Environment page):
 

1. Check if the tkr-vsphere-nonparavirt-v1.25.7_vmware.2-tkg.1 and tkg-vsphere-repository-nonparavir-1.25.7_vmware.2-tkg.1 packages exist in the private Harbor repository.
2. Follow the air gap procedure to download the bundle locally using tanzu isolated-cluster download-bundle --tkg-version v2.2.0 command.
3. Upload the bundle using tanzu isolated-cluster upload-bundle --destination-repo <private-harbor> to your private Harbor.
4.  Copy the leftover packages manually using imgpkg:
   1. imgpkg copy --tar ${source_dir}/tkr-vsphere-nonparavirt-v1.25.7_vmware.2-tkg.1.tar --to-repo <remote-harbor-url>/tkr-vsphere-nonparavirt --registry-ca-cert-path ${harbor-cert-path}/ca.crt
   2. imgpkg copy --tar ${source_dir}/tkr-repository-vsphere-nonparavirt-v1.25.7_vmware.2.tar --to-repo <remote-harbor-url>/tkr-repository-vsphere-nonparavirt --registry-ca-cert-path ${harbor-cert-path}/ca.crt

Additional Information

How to Increase tkr-source-controller deployment resource limits on TKG 2.1.x to resolve 'OOMKilled' Error