During TKGI Cluster Upgrade | error when patching | Invalid value | field is immutable

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Cluster upgrade fails with an error similar to the below at the end of the upgrade while apply-addons errands are running:

\"/var/vcap/jobs/apply-specs/specs/coredns-yaml":error when patching ... Deployment.apps \"coredns\" is invalid: selector: Invalid value: v1.LabeiSelector{MatchLabels:map[string]string\"k8s-app\": "kube-dns\"),MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Environment

• VMware Tanzu Kubernetes Grid Integrated Edition

Cause

• An immutable field was added to the coredns deployment running in the affected cluster.
• apply-addons tries to patch the deployment but fails because the field is immutable.

Resolution

NOTE: This has potential to break any workloads that depend on the custom DNS configurations. 

1. Backup the coredns deployment yaml
   1. Set the kubectl context on the affected cluster
   2. kubectl get deploy coredns -n kube-system -o yaml
2. Pull the core dns deployment yaml from the apply-addons VM of the affected cluster
   1. bosh -d service-instance_<GUID> run-errand apply-addons --keep-alive
   2. bosh -d service-instance_<GUID> ssh <apply-addons_VM>
   3. cat /var/vcap/jobs/apply-specs/specs/coredns-yaml
   4. Copy the deployment section yaml from the output and save it for later.
3. Delete the coredns deployment yaml
   • kubectl delete deploy coredns -n kube-system 
4. Deploy the yaml saved from step 2.4
   • kubectl apply -f <coredns_deployment_yaml_from_2.4> 
5. Re-tigger the cluster upgrade.