Workload Cluster Inoperable: kubectl Commands Fail with "the server has asked for the client to provide credentials" Error After CRD Deletion

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

A workload cluster is rendered inoperable after a user executes the command kubectl delete crd packageinstalls.packaging.carvel.dev.

The user may be able to authenticate to the cluster but attempts to run any kubectl get resources command fail with errors similar to:

E... memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
error: You must be logged in to the server (the server has asked for the client to provide credentials)

This action results in the deletion of all applications and packages from the guest cluster, and multiple critical components (PackageInstalls and Applications) become stuck in a Deleting or Delete failed state.

Cause

The underlying cause is the manual deletion of the packageinstalls.packaging.carvel.dev Custom Resource Definition (CRD).

This CRD is a critical dependency for the cluster's Carvel Packaging component. Removing it corrupts the state of the cluster, prevents the kapp-controller from reconciling packages, and leaves all associated applications and PackageInstalls in a perpetual, unrecoverable terminating state.

Resolution

Please open a Support Request with Broadcom Support