Upgrade to TKGI 1.22.1 fails with error on kube-apiserver ": strict decoding error: unknown field \"metadata\""

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

1.) You upgrade to TKGI v1.22.1 and deploy it via Apply Changes in Opsmanager. This completes but the 'tkgi upgrade-clusters' command fails to upgrade nodes. The master node's status is labeled failing in the 'bosh vms' command output (as shown below):

master/<###-###-###-1234>  failing        AZ1  172.X.xx.x  master_service-instance-<GUID> xlarge   true    bosh-vsphere-esxi-ubuntu-jammy-go_agent/1.844

 ---------------

2.) ssh'ing onto the failing master node/vm identified above will show the 'kube-apiserver' job is failing (as the output of 'monit summary' shows below):

 

Cause

'kube-apiserver' is failing due to Strict decoding rules embedded into the targeted upgrade version (TKGI v1.22.1).  

We can confirm this by ssh'ing into the failing master VM/node and checking the 'kube-apiserver.stderr.log' (located at /var/vcap/sys/log directory).

"command failed" err="error while parsing file: error decoding encryption provider configuration file \"/var/vcap/jobs/kube-apiserver/config/encryption-provider-config\": strict decoding error: unknown field \"metadata\""

--------------

*Notice the above log line is complaining about the "\metadata\" field from the 'encryption-provider-config' (located in the /var/vcap/jobs/kube-apiserver/config/encryption-provider-config\ directory).

 

Resolution

Temporary Solution: 

1. remove any top-level 'metadata' field/block from your 'encryption-provider-config.yaml'
2. run command: 'monit restart kube-apiserver' on the failing vm (requires ssh and sudo privilege).
3. If this above steps do not resolve the issue, please open a Ticket with TANZU Support.

*Please note, the above in only a temporary workaround because any changes will get reverted if the master instance is recreated or upgraded. 

---------------------------

Permanent Solution:

• Modify the cluster's Kubernetes Profile to not include the 'metadata' block in the Encryption Provider Configuration File.  Find below a TKGI reference doc for creating the Profile and Config file:
  • https://techdocs.broadcom.com/us/en/vmware-tanzu/standalone-components/tanzu-kubernetes-grid-integrated-edition/1-22/tkgi/k8s-profiles-encrypt-etcd.html#:~:text=To%20create%20an%20encryption%20provider%20configuration%20file

Additional Information

Note: Enterprise Automation can sometimes add this 'metadata' field back into the config even after manually deleting as the scenario references below:  

1. When creating encryption provider configs for TKGI Kubernetes profiles, and using the kustomize tool, it can automatically add a metadata section to the YAML manifest (encryption-provider-config.yaml).
2. Prior to Kubernetes 1.31.0, this metadata section was ignored and caused no issues
3. Kubernetes 1.31.0 introduced stricter YAML manifest validation that rejects encryption configs with metadata sections.

If you are facing this scenario, then a custom tool to remove/ignore said metadata field will likely be needed to move forward. Contact TANZU Support for further assistance

 

-------------------------

Helpful links:

• https://techdocs.broadcom.com/us/en/vmware-tanzu/standalone-components/tanzu-kubernetes-grid-integrated-edition/1-22/tkgi/k8s-profiles-encrypt-etcd.html#:~:text=To%20create%20an%20encryption%20provider%20configuration%20file
• https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#write-an-encryption-configuration-file