Error: "The apiservices "v1beta1.metrics.k8s.io" is invalid: metadata.resourceVersion: Invalid value: 0x0"

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

• During a TKGi upgrade the apply-addons job fails and the task shows the following:

Stderr   The apiservices "v1beta1.metrics.k8s.io" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update

• ssh'ing to the addons VM after running the keep-alive command flag you find that manually applying the manifests also fails:

1. bosh -d service-instance_xxxx run-errand apply-addons --keep-alive

2. bosh ssh -d service-instance_xxxx <APPLY-ADDONS_VM>

3. sudo -i

4. cd /var/vcap/jobs/apply-specs/specs/metrics-server

5. Copy and attempt to manually apply the manifests in this directory for the metrics apiService to reproduce the error.

Environment

TKGi 1.x deployed environment using metrics server

Cause

When Kubernetes objects are exported manually using kubectl get <OBJECT> -o yaml and then reapplied without removing the resourceVersion or other object revision specific annotations this creates a new resource but with the old resourceVersion. 

The reason behind the failure in TKGi, is that when an apply-addons errand runs it tries to patch the existing resources for metrics server. The addons VM will try patching the existing resources for metric apiService using configs located at /var/vcap/jobs/apply-specs/specs/metrics-server. The kubernetes API will compare the last-applied-configuration with the current yaml being patched, and send the changes. If there is an invalid resourceVersion in last-applied-configuration, it will send the resourceVersion value as null and not replace the old value, even though that resourceVersion chain is no longer valid. This will cause an error making changes to the kubernetes objects and the apply-addons job will fail.

Resolution

1. Login to the cluster using TKGI CLI:
   
   tkgi get-credentials <CLUSTER_NAME>
   
   
2. Backup the current config of the metrics apiService:
   
   kubectl get apiservice v1beta1.metrics.k8s.io -o yaml > metricsapiservce.bak.yaml
   
   
3. Edit the apiService for metrics and remove the annotation kubectl.kubernetes.io/last-applied-configuration and all its contents:
   
    kubectl edit apiservice v1beta1.metrics.k8s.io
   
   
4. Rerun the apply-addon job to ensure the addon succeeded with the new changes:
   
   bosh -d service-instance_xxxx run-errand apply-addons
   
   
5. After the errand task completes successfully, re-run the TKGi cluster upgrade:
   
   tkgi upgrade-cluster <CLUSTER_NAME>