Supervisor Services not reconciling due to "TLS handshake timeout" due to proxy configuration
Article ID: 445481
Updated On:
Products
Issue/Introduction
Multiple Supervisor core services, including tkg.vsphere.vmware.com, Contour, and Harbor, were observed in a ReconcileFailed state.
Several Supervisor PackageInstall (PKGI) objects were also in a ReconcileFailed state. Upon reviewing the PKGI details, the following error was consistently observed:
vendir: Error: Syncing directory '0': Syncing directory '.' with imgpkgBundle contents:
Fetching image: Error while preparing a transport to talk with the registry:
Unable to create round tripper:
Get "https://projects.packages.broadcom.com/v2/":
net/http: TLS handshake timeout
Pods running on the Supervisor Cluster were additionally experiencing image pull failures when attempting to retrieve container images from the public registry.
Network connectivity and name resolution to the registry endpoint were validated from the Supervisor VMs on both the management and workload networks.
DNS resolution, curl, and OpenSSL connectivity tests were successfully completed against the registry endpoint.
Connectivity was specifically verified through both eth0 and eth1 interfaces using curl, and communication with the registry was successful on both paths.
Despite successful DNS resolution and network connectivity validation to the registry from both Supervisor network interfaces, Supervisor core services and PackageInstall resources continued to encounter intermittent image retrieval failures, resulting in TLS handshake timeout errors and subsequent ReconcileFailed states.
Environment
VMware vCenter Server 8.x
vSphere Supervisor service
VMware vSphere Kubernetes Service
Cause
Check the PROXY settings on the vCenter/supervisor level.
A packet capture performed on the Supervisor node (eth0) during communication with projects.packages.broadcom.com showed that the TCP session was reset immediately after the TLS/SSL certificate verification stage. The trace indicated the presence of a TCP Reset (RST) packet, resulting in termination of the TLS handshake before completion.
The Supervisor cluster was configured to inherit proxy settings. If the broadcom.com domain was included in the proxy exclusion (NO_PROXY) list, causing traffic destined for Broadcom registry endpoints to bypass the configured HTTP/HTTPS proxy.
The issue was caused by a proxy bypass misconfiguration. Since the broadcom.com domain was excluded from proxy routing, the kapp-controller attempted to establish a direct outbound connection to projects.packages.broadcom.com instead of using the designated proxy server.
The environment's network security policy requires all outbound internet traffic to traverse the approved HTTP/HTTPS proxy. As a result, the direct internet connection attempt violated the network policy and was blocked by the perimeter firewall/security infrastructure.
During the connection attempt, the firewall intercepted the unauthorized direct outbound traffic and injected a TCP Reset (RST) packet, abruptly terminating the TCP session. This interruption occurred during the TLS handshake process, leading to the observed connectivity failure between the Supervisor nodes and the Broadcom
Resolution
Navigate to the vCenter Server.
Go to Workload Management -> Supervisor Clusters -> Select the affected cluster.
Modify the Network configuration to remove the broadcom.com domain from the proxy exclusion (NO_PROXY) list
Ensure the custom proxy configuration is applied with the correct HTTPS proxy settings (pointing to the HTTPS/HTTP proxy IP).
(OR)
Change the vCenter proxy settings via VAMI and inherit the same to supervisor.
Additional Information
This issue can also happen due MTU mismatch,
Refer: Supervisor Services fail to reconcile due to mgmt-image-proxy LoadBalancer VIP TLS handshake failure
Feedback
