Supervisor Deployment with NSX VPC architecture fails at "Configured Core Supervisor Services" with cci-ns-controller-manager pods in ErrImagePull state
search cancel

Supervisor Deployment with NSX VPC architecture fails at "Configured Core Supervisor Services" with cci-ns-controller-manager pods in ErrImagePull state

book

Article ID: 452173

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

  • When attempting to deploy a vSphere Supervisor using an NSX VPC architecture, the deployment stalls and ultimately fails at the following stage:


  • In the vSphere Client, the Supervisor Management UI shows a warning at Configured Core Supervisor Services with ReconcileFailed and i/o timeout messages.
  • cci-ns-controller-manager pods are stuck in ErrImagePull state after checking on the Supervisor cluster:
    kubectl get pods -n svc-cci-ns-#### | grep cci-ns-controller-manager

svc-cci-ns-####   cci-ns-controller-manager-#####-####   0/1     ErrImagePull          0          11h
svc-cci-ns-####   cci-ns-controller-manager-#####-####   0/1     ErrImagePull          0          19h
svc-cci-ns-####   cci-ns-controller-manager-#####-####   0/1     ErrImagePull          0          3h28m
  • Multiple packages (pkgi) display a Reconcile failed: Error status. 
    kubectl get pkgi -A
    NAMESPACE                           NAME                                      PACKAGE NAME                                         PACKAGE VERSION           DESCRIPTION                                                            AGE   PAUSED
kube-system                         supervisor-authz-service-controller       supervisor-authz-service.vmware.com                  9.1.0-25370922            Reconcile succeeded                                                    20h   
svc-tkg-m7e4b                       runtime-extension                         extension.tkg.vsphere.vmware.com                     3.6.1                     Reconcile failed: Error (see .status.usefulErrorMessage for details)   22h   
svc-tkg-m7e4b                       tanzu-addons-manager                      addons-manager.tanzu.vmware.com                      3.6.1                     Reconcile succeeded                                                    21h   
svc-tkg-m7e4b                       tanzu-auth                                tanzu-auth.tanzu.vmware.com                          3.6.1                     Reconcile succeeded                                                    22h   
svc-tkg-m7e4b                       tanzu-cliplugins                          cliplugins.tanzu.vmware.com                          3.6.1                     Reconcile succeeded                                                    22h   
svc-tkg-m7e4b                       tanzu-cluster-api                         cluster-api.tanzu.vmware.com                         1.12.2+vmware.2           Reconcile succeeded                                                    22h   
svc-tkg-m7e4b                       tanzu-cluster-api-bootstrap-kubeadm       cluster-api-bootstrap-kubeadm.tanzu.vmware.com       1.12.2+vmware.2           Reconcile succeeded                                                    22h   
svc-tkg-m7e4b                       tanzu-cluster-api-control-plane-kubeadm   cluster-api-control-plane-kubeadm.tanzu.vmware.com   1.12.2+vmware.2           Reconcile succeeded                                                    22h   
svc-tkg-m7e4b                       tanzu-cluster-api-provider-vsphere        cluster-api-provider-vsphere.tanzu.vmware.com        1.15.2+vmware.3           Reconcile failed: Error (see .status.usefulErrorMessage for details)   22h   
svc-tkg-m7e4b                       tkg-controller                            gcc.tkg.vsphere.vmware.com                           3.14.1                    Reconcile failed: Error (see .status.usefulErrorMessage for details)   21h   
svc-tkg-m7e4b                       tkgs-config                               tkgs-config-vsphere.tanzu.vmware.com                 3.6.1                     Reconcile failed: Error (see .status.usefulErrorMessage for details)   21h   
svc-tkg-m7e4b                       tkr-service                               tkr-service.tanzu.vmware.com                         3.6.1                     Reconcile succeeded                                                    22h   
svc-tkg-m7e4b                       upgrade-compatibility-service             ucs.tkg.vsphere.vmware.com                           3.6.1                     Reconcile succeeded                                                    21h   
vmware-system-imageregistry         imageregistry-operator                    vmimages.vmware.com                                  9.1.0-25370922            Reconcile succeeded                                                    22h   
vmware-system-supervisor-services   kube-state-metrics                        kube-state-metrics.vsphere.vmware.com                2.14.0-25195109           Reconcile failed: Error (see .status.usefulErrorMessage for details)   20h   
vmware-system-supervisor-services   svc-cci-ns.vmware.com                     cci-ns.vmware.com                                    9.1.0-embedded+739b5075   Reconcile failed: Error (see .status.usefulErrorMessage for details)   22h   
vmware-system-supervisor-services   svc-tkg.vsphere.vmware.com                tkg.vsphere.vmware.com                               3.6.1-embedded+v1.35      Reconcile failed: Error (see .status.usefulErrorMessage for details)   22h   
vmware-system-supervisor-services   svc-velero.vsphere.vmware.com             velero.vsphere.vmware.com                            1.9.0-embedded+25369333   Reconcile failed: Error (see .status.usefulErrorMessage for details)   22h   
vmware-system-workload-cli          workload-cli-plugins                      cliplugins.cli.vmware.com                            9.1.0-0-25296329          Reconcile succeeded                                                    --


  • Pod descriptions and ESXi spherelet logs indicate a failure to resolve the image registry, displaying errors such as:
    failed to get images: Image... has failed. Error: Failed to resolve on node... Reason: Http request failed. Code 400... dial tcp: lookup docker-registry.kube-system.svc: i/o timeout

Environment

VMware vSphere Kubernetes Service

Cause

This issue occurs due to network misconfigurations blocking the Supervisor's ability to download container images from the Broadcom registry (projects.packages.broadcom.com). This is typically caused by one or both of the following:

  • Improper workload network configuration when deploying Supervisor cluster with NSX VPC as Networking as per the following documentation: Workload Network Requirements
  • Infrastructure Blockers (MTU & Routing): The physical switches connecting the ESXi hosts and/or the perimeter firewall have an insufficient Maximum Transmission Unit (MTU) configured. NSX overlay traffic requires an MTU of at least 1600 (typically 1700 for VKS/VPC deployments) to account for encapsulation overhead. Additionally, missing static routes on the firewall can block external egress.

Resolution

To resolve this issue, ensure the network infrastructure is properly configured to support VKS deployment with NSX VPC and the pre-requisites are followed as per the following documentation: Requirements for Supervisor Deployment with VPC Networking

Powered by Wolken Software