CoreDNS pods stuck in ImagePullBackOff state During SSPI (SSP Installer) 5.1 Upgrade

Products

VMware vDefend Firewall with Advanced Threat Prevention VMware vDefend Firewall

Issue/Introduction

After upgrading the SSPI (SSP Installer) from version 5.0 to 5.1, the CoreDNS pods in the kube-system namespace may get stuck in the ImagePullBackOff state. This issue prevents dependent services, such as Authelia, from starting properly, causing them to remain in a CrashLoopBackOff (CLBO) state.

Observed symptoms:

The SSPI upgrade process completes successfully.

The SSP UI becomes inaccessible, as the Authelia pod fails to start and remains in a CrashLoopBackOff state.

The CoreDNS pods in the kube-system namespace continuously fail to start, showing ImagePullBackOff errors.

Running kubectl describe pod coredns-xxxx -n kube-system shows:

Failed to pull image "registry.k8s.io/coredns/coredns:v1.11.1":
rpc error: code = DeadlineExceeded desc = unable to try pulling possible OCI artifact:
get manifest: pinging container registry registry.k8s.io:
Get "https://registry.k8s.io/v2/": dial tcp <public IP>:443: i/o timeout

Environment

SSPI upgrades from version 5.0 to 5.1.

Deployment types:

Air-gapped environments. (No internet access)

Non-air-gapped environments (with internet access) but with strict firewall or zero-trust network policies that restrict access to the registry registry.k8s.io.

Cause

During the Kubernetes upgrade performed as part of the SSP 5.1 upgrade:

kubeadm intentionally preserves the existing CoreDNS configuration to avoid overwriting customizations. However, the CoreDNS deployment manifest continues referencing the older image version:

registry.k8s.io/coredns/coredns:v1.11.1

In certain environments, this results in a pull attempt to the public Kubernetes registry, which fails due to lack of Internet access.

Resolution

(a) SSH to the SSP-I VM using the sysadmin user credentials.

(b) Execute the following command to update the CoreDNS image:

k set image deployment/coredns coredns=registry.k8s.io/coredns/coredns:v1.12.0 -n kube-system

(c) Wait for the SSP UI to become accessible and verify that users can successfully log in.

(d) In SSP UI, wait for cluster heath to be in stable state.