VKS/TKG Cluster Creation Fails or Hangs on Port 6443 in VMware Cloud Foundation Automation
search cancel

VKS/TKG Cluster Creation Fails or Hangs on Port 6443 in VMware Cloud Foundation Automation

book

Article ID: 443073

calendar_today

Updated On:

Products

VCF Automation

Issue/Introduction

When deploying a Tanzu Kubernetes Grid (TKG) or virtual Kubernetes (VKS) cluster within a VMware Cloud Foundation (VCF) Automation organization, the deployment fails or remains stuck indefinitely in a provisioning/failed state. The following specific symptoms are observed:

  • The deployment task has been running or stalled in a "Failed" or "In Progress" status for several days without progressing or automatically recovering.

  • The cluster control plane node successfully registers on the network and responds to basic ICMP ping requests.

  • Network connection attempts (such as telnet or nc) to the Kubernetes API server port (6443) from external networks or management segments fail or time out.

  • The TLS handshake fails or hangs indefinitely when attempting to pull certificates from the control plane node.

  • Kubernetes bootstrap initialization (kubeadm process) fails to complete on the virtual machine nodes.

Environment

  • VMware Cloud Foundation 9.x

Cause

This issue is typically caused by one of two underlying network infrastructure constraints:

  • MTU (Maximum Transmission Unit) Path Mismatch / Packet Dropping: During cluster bootstrap, the control plane node performs kubeadm initialization. This process exchanges large security certificate chains and TLS handshake payloads over TCP port 6443. Since these packets are often transmitted with the "Do Not Fragment" (DF) flag set, any upstream network segment with a smaller MTU size than the packet will silently drop them. If Jumbo Frames (MTU 9000) or overlay-required sizes (minimum MTU 1600 for NSX encapsulation) are not configured consistently across all physical switches, ESXi uplinks, and virtual switches, the TLS handshakes will hang.

  • Perimeter or Distributed Firewall Restrictions: The network security policies or perimeter firewalls are actively dropping or rejecting TCP traffic on port 6443 between the Workload Namespace Network and the Supervisor Ingress Network.

Resolution

To resolve the cluster creation failure, follow the verification and remediation steps detailed below:

Step 1: Verify MTU Settings Across the Infrastructure

Ensure that Jumbo Frames or at least the minimum overlay MTU settings are uniformly configured end-to-end across your environment:

  • Physical Network Switches: Confirm all transit switches are configured for Jumbo Frames (MTU 9000).

  • ESXi Host Uplinks & vDS: Ensure the vSphere Distributed Switches (vDS) and ESXi host uplinks are configured with an MTU of 9000 (or at least 1600).

  • NSX Transport Zones: Verify that the NSX overlay transport zone MTU is configured appropriately (minimum 1600; recommended 9000) to account for geneve/vxlan encapsulation overhead.

Step 2: Test Path MTU for Packet Fragmentation

Log in via SSH to an ESXi host in the cluster and perform a fragmentation-free ping test to ensure large payloads can travel successfully across the host transport network:

# Test with a payload size that verifies standard MTU configurations (e.g., 1473 bytes)
vmkping -d -s 1473 <Target-ESXi-IP>

# If Jumbo Frames are configured, test with a larger payload size (e.g., 8972 bytes)
vmkping -d -s 8972 <Target-ESXi-IP>

  • Analysis: If the vmkping fails or returns a "packet needs to be fragmented" error, there is an MTU mismatch on the physical switches or virtual interfaces that must be addressed by your network engineering team.

Step 3: Validate TCP Port 6443 Connectivity

Verify that port 6443 is open and accessible from a machine situated on the workload network:

# Attempt to open a socket connection to the control plane IP
nc -zv <Control-Plane-IP> 6443
# OR
telnet <Control-Plane-IP> 6443

  • Analysis: If the connection times out, audit your NSX Distributed Firewall (DFW) configurations, Gateway Firewalls, and physical perimeter firewalls. Ensure there are explicit "Allow" rules governing TCP port 6443 traffic moving between the Namespace Network and the Ingress (Supervisor) Network.

Step 4: Validate TLS Handshake and Certificate Chain Delivery

Verify if the TLS handshake can successfully establish and output the certificate chain without hanging:

openssl s_client -showcerts -connect <Control-Plane-IP>:6443

  • Analysis: If this command runs but hangs indefinitely midway through outputting the certificate block, this is a positive confirmation of the MTU mismatch/packet fragmentation drop identified in Step 1 and Step 2.

Step 5: Extract Logs for Advanced Analysis

If all network paths, firewall configurations, and MTU settings have been verified but initialization continues to fail:

  1. Access the console of the control plane node.

  2. Retrieve and inspect the cloud-init logs:

    cat /var/log/cloud-init-out.log

  3. Generate a diagnostic log bundle from VCF Automation (VCFA) and open a support request with Broadcom Support for deeper code-level diagnostics.

Powered by Wolken Software