Error: ProgressDeadlineExceeded message: ReplicaSet "capi-kubeadm-bootstrap-controller-manager" has timed out progressing during TKGS/VKS version upgrade

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

• Updating the TKGS/VKS version hangs with the below error:
  Configured Core Supervisor Services
Service: tkg.vsphere.vmware.com. Reason: ReconcileFailed. Message: kapp: Error: waiting on reconcile packageinstall/tanzu-cluster-api-bootstrap-kubeadm (packaging.carvel.dev/v1alpha1) namespace: svc-tkg-domain-####: Finished unsuccessfully (Reconcile failed: (message: kapp: Error: waiting on reconcile deployment/capi-kubeadm-bootstrap-controller-manager (apps/v1) namespace: svc-tkg-domain-#####: Finished unsuccessfully (Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "capi-kubeadm-bootstrap-controller-manager-##########" has timed out progressing.)))).
Service: velero.vsphere.vmware.com. Status: Running


• The failure is accompanied by a pod stuck in a Pending state with scheduling events referencing unavailable host ports.
  Warning FailedScheduling MmSSs (x## over ##d) default-scheduler 0/# nodes are available: # node(s) didn't have free ports for the requested pod ports, # node(s) didn't match Pod's node affinity/selector. preemption: 0/# nodes are available: # No preemption victims found for incoming pod, # Preemption is not helpful for scheduling.
   
• Running the following command shows an extra port in use for 8085:
  # kubectl get pods -o yaml -A | grep -i hostport | sort | uniq -c | grep -E '9875|9441|8085'
      4         hostPort: 8085
      3         hostPort: 9441
      3         hostPort: 9875

Environment

• Tanzu Kubernetes Grid Service
• vSphere Kubernetes Service

Cause

• Both the capi-kubeadm-bootstrap-controller-manager and velero.vsphere.vmware.com Supervisor Services attempt to bind to hostPort 8085. Since hostPort is a node-level resource, only one pod per node claims a specific hostPort.
• Although the capi-kubeadm deployment is configured for two replicas, a third pod enters a Pending state during rollout, likely due to a transient update or restart triggering a temporary additional pod. Velero occupies hostPort 8085 on one node, and the running capi-kubeadm pods use the other two nodes, leaving no node available to schedule the third pod. This results in a scheduling deadlock that blocks the deployment and causes the upgrade to stall.

Resolution

1. Temporarily scale down the Velero Supervisor Service to release hostPort 8085.
   kubectl scale deploy/backup-driver -n velero --replicas=0


2. Once the port is freed, the stuck capi-kubeadm pod is scheduled, and the upgrade proceeds. After the rollout completes and the deployment stabilizes with only two replicas, scale Velero back up to its desired state. It lands on the third node where hostPort 8085 is no longer in use, avoiding further conflict.
   kubectl scale deploy/backup-driver -n velero --replicas=1