Resource quota controller stopped updating "status.used" after upgrading TKGm cluster to kubernetes v1.32

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

After upgrading the TKGm cluster to Kubernetes 1.32 (v1.32.5+vmware.1), the resource quota controller stopped updating "status.used", causing the quota to report incorrect pod counts and blocking deployments even when actual usage was well below the limit.

Investigation of the kube-controller-manager logs revealed repeated warnings occurring 4 times every ~15 seconds.

The watchlist request for pods ended with an error, falling back to the standard LIST semantics, err = ListOptions.meta.k8s.io "" is invalid: sendInitialEvents: Forbidden: sendInitialEvents is forbidden for watch unless the WatchList feature gate is enabled 

Unlike other controllers (e.g. replicaset, deployment) which recover correctly via the LIST/WATCH fallback, the quota controller does not trigger a re-sync of status.used after the fallback, which appears to be a regression introduced in 1.32 related to the WatchList feature gate being enabled in the controller manager but not on the API server.

Environment

TKGm 2.5.4

Kubernetes version v1.32.5+vmware.1

Cause

The feature gate [watchlist, watchlistclient] was in beta until version 1.31 and is enabled by default from Kubernetes 1.32 and higher.

In Kubernetes version 1.32, communication is divided into two primary parts:

• The Server (Kube API server holding cluster data)
• The Client (Kube Controller Manager that tracks resource usage and pod counts)

By default:
The Client (Kube Controller Manager) in version 1.32 automatically attempts to use a new, advanced data-streaming protocol to read logs and metrics.
The Server (Kube API Server) is explicitly configured to block this protocol for security and performance reasons. Normally, the client should gracefully revert to standard methods when blocked.

Resolution

Workaround for the issue is, by applying --feature-gates=WatchListClient=false, you instruct the client to never request this streaming protocol, completely bypassing the broken error loop.

For  Plan-Based (Legacy) Clusters:

1. Switch your active command line profile to your TKGm Management Cluster.
2. Locate your cluster's declarative control plane object using:
   • kubectl get kubeadmcontrolplane -A
3. Open the control plane object for editing (replace with your cluster name and namespace):
   • kubectl edit kubeadmcontrolplane <workload-cluster-name>-control-plane -n <cluster-namespace>
4. Update the spec.kubeadmConfigSpec.clusterConfiguration block to append the explicit extra argument structure and save the file 

spec:
kubeadmConfigSpec:
clusterConfiguration:
controllerManager:
extraArgs:
feature-gates: WatchListClient=false

Cluster API (CAPI) will perform a rolling restart of your control plane elements.