VKS Cluster Management Service Stuck in "Busy" State in Air-Gapped Environments

Products

• VMware vSphere Kubernetes Service
• VMware Cloud Foundation

Issue/Introduction

• The vSphere Kubernetes Service (VKS) Cluster Management service continuously reports a "Busy" status in the VMware Cloud Foundation Automation UI.
• Lifecycle management operations on VKS clusters are blocked or fail to initiate.
• Reviewing the cluster agent logs reveals image pull timeout errors for the cluster agent pod in VMware Cloud Foundation Automation:

kubectl logs cluster-agent-service-server-##### -n prelude

Failed to pull image "projects.packages.broadcom.com/vsphere/vksm/extensions/9.0.2-0-25145732/tmc-bootstrapper/manifest:latest":

rpc error: code = Unknown desc = failed to pull and unpack image: failed to resolve reference:

failed to authorize: failed to fetch anonymous token: Get "https://projects.packages.broadcom.com/v2/token": dial tcp: lookup projects.packages.broadcom.com: i/o timeout

Environment

vSphere Kubernetes Service (VKS) 

VMware Cloud Foundation (VCF) 9.x

Cause

This issue occurs when VKS Cluster Management is deployed in an air-gapped or internet-restricted network topology.

By default, the tmc-bootstrapper extension pod attempts to pull mandatory component manifest bundles directly from Broadcom’s public endpoint (projects.packages.broadcom.com). In an isolated environment without access to the internet, these outbound image pull requests time out indefinitely. As a result, the initialization pipeline cannot complete, leaving the service locked in a "Busy" state.

Resolution

To resolve this issue, configure the Supervisor to fetch extension manifests from an internal private container registry, or establish a Supervisor Management Proxy for outbound connectivity.

Option 1: Configure Air-Gapped Image Location Mapping (Recommended)

1. Ensure the required tmc-bootstrapper and VKS extension image bundles are mirrored to your local private container registry (e.g., Harbor).
2. Update the Supervisor configuration to map extension manifest requests to your local registry endpoints rather than the default public URLs.
3. For steps 1 and step 2, refer to the detailed steps in the Broadcom Documentation:

👉 Enabling VKS Cluster Management in an Air-Gapped Scenario

Option 2: Configure Supervisor Management Proxy

If your architecture routes external traffic through an enterprise HTTP/HTTPS proxy rather than a purely local registry mirror:

1. Configure the Supervisor network settings to route management plane outbound traffic through your corporate proxy server.
2. Verify that projects.packages.broadcom.com is whitelisted on your proxy server over TCP port 443.
3. Follow the implementation guide here:

👉 Supervisor Management Proxy Configuration