Upgrading Tanzu Kubernetes Grid (TKG) from TKG 1.2.1 to TKG 1.3.0 fails with 'unable to get cluster configuration: unable to get template'

Products

• Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:

You have a Tanzu Kubernetes Grid (TKG) v1.2 management cluster created in an internet-restricted environment.

You are trying to upgrade the management cluster to TKG v1.3.

You are using a custom container registry.

You are using the latest tanzu CLI v1.3.0 to upgrade with the commands below: 

export TKG_CUSTOM_IMAGE_REPOSITORY="my.custom.repo"
tanzu management-cluster upgrade

The upgrade fails with:

Error: failed to deploy additional components after kubernetes upgrade: unable to get cluster configuration: unable to get template: Overlaying (in following order: overlay.yaml, 01_plans/dev.yaml, 02_addons/cni/add_cni.yaml, 03_customizations/annotate_os_info.yaml, 03_customizations/filter.yaml, 03_customizations/registry_ca_cert.yaml, 03_customizations/registry_skip_tls_verify.yaml): Document on line 03_customizations/registry_ca_cert.yaml:7: Expected number of matched nodes to be 1, but was 0

Cause

The issue occurs with the ytt overlays when the tanzu CLI  needs to read specific ytt files.

It does this in order to deploy additional components to the management cluster.

The ytt cluster templates are processed in lexicographical order. When processing ytt templates, the  existing ytt/03_customizations/filter.yaml file needs to be processed last.

However, the current ytt/03_customizations/filter.yaml file is processed before the ytt/03_customizations/registry_ca_cert.yaml and ytt/03_customizations/registry_skip_tls_verify.yaml files.

Resolution

To resolve this issue, update the filename of ytt/03_customizations/filter.yaml so that it is processed last.

This is done by renaming the file to: ytt/03_customizations/zfilter.yaml.

Run the following command:

mv ~/.tanzu/tkg/providers/ytt/03_customizations/filter.yaml ~/.tanzu/tkg/providers/ytt/03_customizations/zfilter.yaml

Then re-run the upgrade:

export TKG_CUSTOM_IMAGE_REPOSITORY="my.custom.repo"
tanzu management-cluster upgrade

Workaround: