Diagnosing Intermittent TLS Handshake Failures to Internal Service VIPs - VMware vSphere Kubernetes Service

Products

• VMware NSX

Issue/Introduction

This article provides guidance on troubleshooting intermittent connection failures and timeouts when pods or proxies attempt to communicate with internal Kubernetes Service VIPs (e.g., #.#.#.#) within the Supervisor cluster.

• Pods or internal proxies (e.g., masterproxy) experience intermittent connection failures or timeouts when attempting HTTPS communication to internal Service VIPs.
• TLS handshakes hang or reset consistently.
• Packet captures show drops for large packets (e.g., 1500+ bytes) occurring at the VRRP gateway or host switchport level.
• Error messages include upstream timed out or Connection reset by peer during SSL/TLS handshaking.

Environment

• VMware vSphere Kubernetes Service (VKS)
• NSX-T / NSX Data Center (Geneve Overlay)
• Supervisor Cluster

Cause

Intermittent TLS handshake failures to internal Service VIPs in a VKS Supervisor cluster are frequently caused by MTU mismatch issues or packet fragmentation across the NSX Geneve overlay network. 

Resolution

To diagnose and resolve, follow these steps:

1. Perform a packet capture from the affected node (e.g., Control Plane VM or ESXi host) while reproducing the connection issue.
2. Analyze the packet capture for dropped packets specifically during the TLS Client Hello exchange. Large TLS packets are more likely to be dropped if the fabric MTU is misconfigured.
3. Verify that the physical fabric (switch/router) supports Jumbo Frames (1600+ bytes) across all host-to-host Geneve tunnel endpoints.
4. Use the following command to test direct pod-to-pod overlay routing, bypassing the Distributed Load Balancer (DLB) VIP: curl -kv https://#.#.#.#:#
5. Inspect Distributed Firewall (DFW) rules, stateful NAT entries, and OVS flow tables on the impacted host switchports.
6. Verify that no OVS flow table drops or firewall policies are blocking the TLS handshake.

Additional Information

For detailed instructions on capturing network traffic for analysis, review the documentation on retrieving support bundles and logs. To speak with a customer representative or a Support Engineer, see Contact Support. Scroll to the bottom of the page and click on your respective region.