Supervisor Pod 'cci-ns-controller-manager' Stuck in 'ErrImagePull' Loop with 'RPC

Products

VMware vSphere Kubernetes Service VMware Cloud Foundation

Issue/Introduction

When deploying Supervisor Cluster environment, the system-defined namespace management pod cci-ns-controller-manager fails to initialize. The pod replicas remain trapped in an ErrImagePull or permanent Pending state, stalling configuration convergence and restricting downstream control plane operations.

Infrastructure administrators will observe distinct error footprints across both the Kubernetes API layer and the localized hypervisor container runtimes:

Kubernetes Deployment Status

Checking the workload framework inside the affected infrastructure namespace reveals degraded replica states with zero active availability:

kubectl get pods -n <target-namespace>

Output Example:

NAME READY STATUS RESTARTS AGE

cci-ns-controller-manager-#####-##### 0/1 ErrImagePull 0 172m

cci-ns-controller-manager-#####-##### 0/1 ErrImagePull 0 3d

cci-ns-controller-manager-#####-##### 0/1 ErrImagePull 0 4h52m

masterproxy-cci-ns-plugin-##### 1/1 Running 0 3d

2. Kubelet Event Log Tracking

Inspecting the granular properties of the failing pod exposes a critical infrastructure-layer failure rather than a routine network timeout or standard registry credential mismatch:

kubectl describe pod <failed-pod-name> -n <target-namespace>

Log Signature:

Warning Failed 54m kubelet (combined from similar events): failed to get images: Image <target-namespace>/vkm-svs-<image-hash> has failed. Error: Failed to resolve on node <esxi-worker-node>. Reason: Failed to exec /bin/crx-cli run: failed to execute cmd. RC=1, Stderr=CrxCli_InstanceCreate:1043: RPC_BootstrapPowerOn Failed to prepare the VM to boot Linux. See the virtual machine's log file for more details.

Module 'DirectBootLinux' power on failed.

Failed to start the virtual machine.

Environment

VMware Cloud Foundation 9.x

Cause

This failure occurs because the Kubernetes workload scheduler incorrectly assigns the system management pod to a standard ESXi worker compute node instead of reserving it exclusively within designated, specialized control plane instances.

In a vSphere Supervisor cluster architecture, management pods must execute inside secure, highly optimized, known as Pod VMs. When the scheduler mistakenly places the workload onto an ESXi worker node residing within an isolated or heavily firewalled network segment, that host lacks the active routing tables or ingress paths required to reach the internal core management registry (`docker-registry.kube-system.svc`).

Because the host-level container client (`crx-cli`) cannot access the registry endpoint to pull and extract the necessary base image layers, the hypervisor's`DirectBootLinux`driver module instantly drops the execution loop. This triggers the `RPC_BootstrapPowerOn Failed` exception and prevents the underlying pod from booting cleanly.

Resolution

To restore operational availability, a localized containment strategy must be executed to isolate the unrouted host and redirect scheduling parameters back onto fully synchronized control-plane compute resources.

Step 1: Isolate the Affected ESXi Host

Apply a structural cordon to the ESXi host flagged in the pod's event log. This systematically blocks the scheduling engine from routing subsequent namespace operations into this isolated network path:

kubectl cordon <affected-esxi-node-name>

Note: Nodes are cluster-scoped resources. Do not append the -n namespace flag, as it is automatically ignored by the API server.

Step 2: Evict and Reschedule the Degraded Workloads

Purge the non-viable pod instances. This forces the active replication controller to re-evaluate placement matrices and generate fresh scheduling requests:

kubectl delete pod <failed-pod-name> -n <target-namespace>

Step 3: Verify Infrastructure Convergence

Monitor the namespace state to ensure that the replication manager instantiates the new cci-ns-controller-manager workloads directly onto dedicated management-plane hosts that possess fully synchronized network paths to the management registry:

kubectl get pods -n <target-namespace> -w

Expected Result: The newly generated pod successfully reaches the internal registry endpoint, the DirectBootLinux kernel initialization executes flawlessly, and the pod transitions cleanly into a stable Running state.

Step 4: Long-Term Topology Correction (Optional)

If the cordoned ESXi host must accept core Supervisor management workloads in the future, update your physical/virtual network architecture to ensure that the host's management interface has an unhindered routing path to the docker-registry.kube-system.svc segment. Once network parity is verified, rejoin the host to the active scheduling pool:

kubectl uncordon <affected-esxi-node-name>