VKS 3.6 workload cluster rollout stuck in Deleting phase after applying TuneD / osConfiguration changes

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

When applying a TuneD profile through the VKS osConfiguration ClusterClass variable at the cluster level, the workload cluster rollout becomes stuck.

During the rollout, you may observe the following symptoms:

• A worker Machine becomes stuck in the Deleting phase (initially reporting WaitingForVolumeDetach, later showing SuccessfulDrainNode).

• One or more control plane nodes become unhealthy and report etcd-related errors.

• The workload cluster API becomes degraded, causing commands such as kubectl get ns to intermittently fail with the server has received too many requests and has asked to try again later.

The containerd logs (journalctl -u containerd) show the service failing and exiting:

failed: error=rpc error: NotFound desc = an error occurred when try to find container: not found
skip loading plugin error: failed to check mkfs.erofs availability: failed to run mkfs.erofs executable file not found in $PATH
failed to detect task error: rpc error container not created
cleaning leaked shim process
failed to detect task error: rpc error container not created
shim disconnected
cleaning up after shim disconnected
cleaning up dead shim
Stopping containerd container runtime
containerd: context cancelled
containerd.service: Main process exited, code=exited, status=1/FAILURE
failed with result exit-code

The kubelet logs (journalctl -u kubelet) report PLEG health issues and a down runtime:

status from runtime service failed... rpc error
container runtime sanity check failed
failed to get pod or container map
container runtime sanity check failed
failed to get pod or container map
PLEG is not healthy
rpc error
runtime service failed
Container runtime is down
PLEG is not healthy
PLEG was last seen active ...

Environment

 

• VMware vCenter 8.0 Update 3

• VMware Kubernetes Service (VKS) 3.6

• ClusterClass: builtin-generic-v3.6.0

• Argo CD / Argo Rollouts active on the cluster

 

Cause

The custom TuneD profile (e.g., core-dump-to-pvc-v1) modifies kernel and OS parameters related to core dumping or storage management. When this OS-level configuration change occurs simultaneously with Argo CD/Rollouts aggressively orchestrating a high volume of container lifecycles, containerd encounters unhandled states. This combination causes containerd to panic, drop its shims, and exit with status=1/FAILURE.

Resolution

To recover an actively stuck rollout, restart the containerd service on the affected nodes.

To safely apply a TuneDProfile to the cluster without encountering this issue, follow these steps:

1. Scale Argo CD deployments down to zero replicas to pause high-volume container lifecycle orchestration.

2. Apply the TuneDProfile (or osConfiguration change) to the cluster.

3. Monitor the environment and allow the worker node rollout to complete successfully.

4. Scale Argo CD deployments back up to their original replica counts.