Remove wrongly configured proxy.systemWide variable before upgrading TKG on AWS or Azure from 2.2 to 2.3

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:

You will see the following error messages and the upgrade will fail if they:

• You are using TKG on AWS and Azure
• Configured  proxy setting
• Trying to upgrade from TKG 2.2.0 to TKG 2.3.0 

18:23:59  Applying patch to resource tkg-up-minor-4-mgmt-aws of type *v1beta1.Cluster ...   18:23:59  error while applying patch for "&TypeMeta{Kind:,APIVersion:,}" tkg-system/tkg-up-minor-4-mgmt-aws: admission webhook "default.cluster.cluster.x-k8s.io" denied the request: Cluster.cluster.x-k8s.io "tkg-up-minor-4-mgmt-aws" is invalid: spec.topology.variables: Invalid value: "": failed validation: "proxy.systemWide" fields are not specified in the variable schema of variable "proxy", retrying   18:24:32  error while applying patch for "&TypeMeta{Kind:,APIVersion:,}" tkg-system/tkg-up-minor-4-mgmt-aws: admission webhook "default.cluster.cluster.x-k8s.io" denied the request: Cluster.cluster.x-k8s.io "tkg-up-minor-4-mgmt-aws" is invalid: spec.topology.variables: Invalid value: "": failed validation: "proxy.systemWide" fields are not specified in the variable schema of variable "proxy", retrying

Environment

VMware Tanzu Kubernetes Grid 1.x

Cause

This is caused by wrongly exposing the proxy.systemWide default value to AWS and Azure cluster variables when configuring proxy-related variables in TKG on AWS and Azure.

Resolution

Removing the wrongly populated proxy.systemWide variable and its value can solve the problem.

Additional Information

Customers can manually edit the cluster object to remove the spec.topology.variables/proxy/value/systemWide variable.

Or they can run the following short script in the management cluster kubeconfig context to remove systemWide variable in all cluster objects( jq should be installed on the jumper machine running following script):

 

#!/bin/bash clusters=$(kubectl get cluster -A --no-headers | awk '{print $1 "/" $2}') for cluster in $clusters do   cluster_ns="${cluster%/*}"   cluster_name="${cluster#*/}"   index=$(kubectl get cluster ${cluster_name} -n ${cluster_ns} -o json  | jq '.spec.topology.variables | map(.name == "proxy") | index(true)')   kubectl patch cluster ${cluster_name} -n ${cluster_ns} --type=json -p='[{"op": "remove", "path": "/spec/topology/variables/'${index}'/value/systemWide"}]' done