VKS Workload Cluster upgrade fails with error message "error reconciling the Cluster topology: failed to create patch helper for Cluster"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• During a workload cluster upgrade, the process fails to progress, leaving Machines and Nodes on their source versions
• On describing cluster, displays the following error message:
  
  kubectl describe cluster <cluster-name> -n <cluster-namespace>
  
  Message: error reconciling the Cluster topology: failed to create patch helper for Cluster <cluster-namespace/cluster-name>: server side apply dry-run failed for modified object: failed to prune fields: failed add back dangling items: failed to convert pruned object to last applied version: conversion webhook for cluster.x-k8s.io/v1beta1, Kind=Cluster failed: internal error occurred during conversion'
  
  
• The capi-controller-manager logs shows below error:
  
  kubectl logs capi-controller-manager-#### -n svc-tkg-domain-c8

YYYY-MM-DD:T:HH:MM:SS stderr F E0220 06:01:59.830079       1 conversion.go:80] "failed to convert" err="internal error occurred during conversion" logger="conversion-webhook" request=""
YYYY-MM-DD:T:HH:MM:SS stderr F E0220 06:01:59.834224       1 controller.go:474] "Reconciler error" err="error reconciling the Cluster topology: failed to create patch helper for Cluster <cluster-namespace/cluster-name> : server side apply dry-run failed for modified object: failed to prune fields: failed add back dangling items: failed to convert pruned object to last applied version: conversion webhook for cluster.x-k8s.io/v1beta1, Kind=Cluster failed: internal error occurred during conversion" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="<cluster-namespace/cluster-name> reconcileID="..."
YYYY-MM-DD:T:HH:MM:SS stderr F E0220 06:01:59.989971       1 runtime.go:142] "Observed a panic" panic="runtime error: index out of range [0] with length 0" panicGoValue="runtime.boundsError{x:0, y:0, signed:true, code:0x0}" stacktrace=<
YYYY-MM-DD:T:HH:MM:SS stderr F         goroutine 1594404 [running]:
YYYY-MM-DD:T:HH:MM:SS stderr F         k8s.io/apimachinery/pkg/util/runtime.logPanic({0x55fbbbc69858, 0xc00416f450}, {0x55fbbba903a0, 0xc003667a58})
YYYY-MM-DD:T:HH:MM:SS stderr F                 /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:132 +0xbc
YYYY-MM-DD:T:HH:MM:SS stderr F         sigs.k8s.io/controller-runtime/pkg/webhook/conversion.(*webhook).handleConvertRequest.func1()
YYYY-MM-DD:T:HH:MM:SS stderr F                 /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/conversion/conversion.go:102 +0xa6
YYYY-MM-DD:T:HH:MM:SS stderr F         panic({0x55fbbba903a0?, 0xc003667a58?})
YYYY-MM-DD:T:HH:MM:SS stderr F                 /usr/local/go/src/runtime/panic.go:792 +0x132
YYYY-MM-DD:T:HH:MM:SS stderr F         sigs.k8s.io/cluster-api/api/core/v1beta1.(*Cluster).ConvertTo(0xc0052e9340, {0x55fbbbc5b530?, 0xc0007e8008})
YYYY-MM-DD:T:HH:MM:SS stderr F                 /go/src/sigs.k8s.io/cluster-api/api/core/v1beta1/conversion.go:78 +0x607

Environment

VMware vSphere Kubernetes Service 3.6.0

VMware vSphere Kubernetes Service 3.6.1

Cause

The issue occurs as kube-apiserver sends incomplete conversion requests to the CAPI controller when attempting to convert managedFields which causes the controller to panic.

Resolution

This issue is resolved in VMware vSphere Kubernetes Service 3.6.2. Upgrade the service using the following steps:

• In the vSphere Client, navigate to Supervisor Management > Services.
• On the vSphere Kubernetes Service tile, select Actions > Install on Supervisors.
• Select version 3.6.2 to perform the upgrade.

Download the VKS 3.6.2 Supervisor Service from the Broadcom Support Portal if it is unavailable in your environment.

Additional Information