Guest Clusters in TMC Self-Managed Fail to Reconcile Packages with "Connection Refused" Error to extensions.stacks.whitesand.tmc.tanzu.broadcom.com
search cancel

Guest Clusters in TMC Self-Managed Fail to Reconcile Packages with "Connection Refused" Error to extensions.stacks.whitesand.tmc.tanzu.broadcom.com

book

Article ID: 429445

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

In Tanzu Mission Control (TMC) Self-Managed environments, managed packages (such as cert-manager, contour, external-dns, etc.) may fail to reconcile or update on Guest Clusters. When inspecting the PackageInstall (pkgi) status or kapp-controller logs, the following connection error is observed:

vendir: Error: Syncing directory '0': Syncing directory '.' with imgpkgBundle contents: Fetching image: Error while preparing a transport to talk with the registry: Unable to create round tripper: Get "https://extensions.stacks.whitesand.tmc.tanzu.broadcom.com/v2/": dial tcp [IP]:443: connect: connection refused

or 

vendir: Error: Syncing directory '0':
  Syncing directory '.' with imgpkgBundle contents:
    Fetching image:
      Error while preparing a transport to talk with the registry:
        Unable to create round tripper:
          Get "https://extensions.stacks.blacksea.tmc.tanzu.broadcom.com/v2/":
            Service Unavailable

This issue typically impacts environments migrated from TMC SaaS to TMC Self-Managed (Nov/Dec 2025 timeframe). While clusters may have appeared healthy initially, failures began occurring globally following the decommissioning of the legacy Broadcom "whitesand" and "blacksea" infrastructure in February 2026. Where whitesand and blacksea refers to different regions.

Environment

Tanzu Mission Control Self Managed 1.4.x

Cause

The failure occurs when a Guest Cluster is unable to successfully reconcile its assigned local package repository, leading the system to fall back to a legacy hardcoded default.

  1. The FQDNs extensions.stacks.whitesand.tmc.tanzu.broadcom.com and extensions.stacks.blacksea.tmc.tanzu.broadcom.com served as the repository endpoints for TMC SaaS. This infrastructure was decommissioned and removed in February 2026.
  2. Any disruption that prevents the Guest Cluster from reaching or trusting the local private registry will cause the repository reconciliation to fail. This is commonly caused by:
    • Missing or unreachable Harbor CA certificates (e.g., failure of the /api/v2.0/systeminfo/getcert endpoint).
    • Firewall or routing changes preventing the cluster from reaching the internal registry FQDN.
    • Expired or missing imagePullSecrets required to access the private registry.
  3. Within the TMC Self-Managed codebase, if the local repository sync for individual clusters fails and the TMC SM cluster was provisioned with incorrect defaults, the system falls back to a hardcoded legacy endpoint. In some cases, if missed during SaaS to SM migration, this fallback is the now-offline "whitesand" path.

Because the "whitesand" and "blacksea" URLs are now decommissioned, this fallback results in a connection refused error rather than a successful (but deprecated) package fetch.

Resolution

To resolve this, the communication break to the local registry must be repaired, and the TMC SM deployment configuration must be updated to ensure clusters are pointed to the correct internal repository.

Step 1: Relocate Package Bundles

Confirm that all required Tanzu Standard package bundles have been successfully mirrored to your private registry.

Step 2: Validate and Restore Local Registry Connectivity

  1. In the TMC UI, verify if the Tanzu Repositories for the affected cluster(s) are in a Succeeded or Failed state.
  2. If the status is Failed, troubleshoot and resolve the underlying blocker to ensure the repository reaches a Succeeded state. High-level areas to investigate include:
    1. Ensuring registry CAs are correct.
    2. Verifying network line-of-sight and firewall permissions.
    3. Confirming all images were successfully pushed to the registry (see step 1).
    4. Checking for typos or invalid paths in the configured repository URLs.

Step 3: Audit and Update TMC SM Deployment Values

Update the default repo in TMC SM to ensure it pushes the correct registry paths to all managed clusters.

  1. Review the values.yaml used for the TMC SM installation.
  2. Check tanzuStandard.relativePath and imageRegistry:
    • If these keys are removed or unset, the default repository fallback is disabled.
    • Update these keys to point explicitly to your local Harbor registry or the new online Broadcom repo (projects.packages.broadcom.com).
  3. Apply the update:
    1. tanzu package installed update tanzu-mission-control -p tmc.tanzu.vmware.com --version 1.4.1 --values-file values.yaml --namespace tmc-local

Step 4: Force Metadata Refresh on Guest Clusters

Existing clusters may have cached the invalid "whitesand" or "blacksea" metadata.

  1. Restart kapp-controller pods on affected Guest Clusters:
    • kubectl rollout restart deployment kapp-controller -n tkg-system
Powered by Wolken Software