Supervisor cluster shows Unhealthy in TMC and "Installation failed" in vCenter GUI due to firewall

Products

• VMware Tanzu Mission Control
• VMware vSphere with Tanzu

Issue/Introduction

• A previously registered and healthy Supervisor cluster becomes unhealthy and in the vCenter GUI, on the Supervisor -> Configure -> Tanzu Mission Control page, shows "Installation failed".
• When reviewing the Supervisor cluster pods, all system pods are in a healthy state. The Supervisor Cluster is healthy.
• The user might see the "agentupdater-workload" pod in the svc-tmc-c# namespace reporting ImagePullBackOff state.
• When reviewing journalctl logs on the Supervisor node on which the agentupdater-workload pod is failing, kubelet logging might report:
  
  

  Nov 05 02:09:01 ##########ca7656ba3109########## containerd[62105]: time="2024-11-05T02:09:01.056445556Z" level=info msg="trying next host" error="failed to do request: Head \"https://<TMC_CLOUD_EXTENSION_URL>/v2/extensions/agent-updater/agentupdater-workload/manifests/sha256:ee1bac1591d45a0f2fd43466651977c9999500539c18845f3dc793e9a8900e7c\": dial tcp ###.###.###.###:443: connect: connection refused" host=<TMC_CLOUD_EXTENSION_URL>

Environment

This failure condition is not version specific. It might impact any version of vSphere with Tanzu connected to TMC.

Cause

This failure condition is caused by a firewall blocking connectivity to the <TMC_CLOUD_EXTENSION_URL> reported in the journalctl logging from the Supervisor control plane node.

Resolution

Review firewall policies to ensure port 443 is opened from the Supervisor cluster to the TMC URL's in order to allow image pulls. This document details the requirement. 

 

If a proxy is in use on the Supervisor Cluster, or configured in TMC. Users might need to investigate connectivity from the Supervisor node through the Proxy to the TMC endpoints to ensure successful connections.