Troubleshooting svc-auto-attach Pod ErrImagePull and TLS Handshake Timeout
search cancel

Troubleshooting svc-auto-attach Pod ErrImagePull and TLS Handshake Timeout

book

Article ID: 437963

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

The svc-auto-attach package fails to reconcile. You may observe the following:

  • The auto-attach pod is stuck in an ErrImagePull or CrashLoopBackOff state.

  • Checking the Image CR or pod events shows the following error:

    Failed to resolve on node <hostname>. Reason: Http request failed. Code 400: ErrorType(2) failed to do request: Head "https://mgmt-image-proxy.kube-system.svc.cluster.local/...": net/http: TLS handshake timeout

Environment

VMware vSphere Kubernetes Service

VMware Cloud Foundry 9.x

Cause

This issue can be caused by various reasons which could interfere end-end IP communications.

- MTU Mismatch
- No route between endpoint
- Misconfiguration in underlying Networking layer
- Firewall Settings Blocking the communication

For example, in this specific case it is caused by an MTU mismatch in the physical network or on the ESXi VMkernel ports.

Overlay networking (VXLAN/Geneve) requires Jumbo Frames (MTU 1600+). While small packets might pass successfully, large packets (like those generated during a TLS certificate exchange for an image pull) will be fragmented or dropped if any physical switch or VMkernel adapter in the path is restricted to a standard MTU of 1500. This results in the TLS handshake timeout.

Resolution

Step 1: Check Supervisor & NSX Layer

Before isolating the ESXi hosts, rule out Control Plane or Firewall issues:

  • Check Pod Events: Run kubectl describe pod <pod-name> -n <namespace> to look for CNI errors or confirm the TLS timeout.

  • Check NCP Logs: Review the vmware-system-netop namespace logs for IPAM exhaustion (e.g., "Insufficient IP capacity in the subnet").

  • NSX Traceflow / DFW: Use the NSX Traceflow UI to ensure a Distributed Firewall "Default Drop" rule isn't blocking the traffic before blaming MTU.

Step 2: Verify VMkernel Port MTU Settings

Ensure the ESXi interfaces handling overlay traffic (typically vmk10 and vmk11) and management (vmk0) are configured for Jumbo Frames (9000). Run the following on the ESXi host:

esxcfg-vmknic -l

Check the MTU column to ensure the appropriate interfaces are set to 9000 rather than 1500.

Step 3: vmkping MTU Sweep to NSX Edge TEPs

Identify your NSX Edge TEP IPs and run a ping test from the ESXi host's VXLAN VMkernel port to verify if large packets are being dropped in the physical network.

Note: Standard tools like nc -vz or curl without large payloads might succeed because small packets do not exceed the 1500 MTU limit. A TLS handshake exchanges certificates, creating large packets that exceed the standard MTU once overlay encapsulation (VXLAN/Geneve) is added. This is why testing with the -s 1572 and -d (Do Not Fragment) flags is mandatory.

  • Test with Standard MTU (1472 payload + 28 byte header = 1500 MTU):

    vmkping -S vxlan -s 1472 -I vmk10 -d <NSX_EDGE_TEP_IP>
    

    (If this succeeds, basic routing is working).

  • Test with Jumbo Frames (1572 payload + 28 byte header = 1600 MTU):

    vmkping -S vxlan -s 1572 -I vmk10 -d <NSX_EDGE_TEP_IP>
    

    (If this fails with 100% packet loss, an upstream physical switch is dropping the jumbo frames).

Step 4: Packet Capture (ESXi to NSX Edge)

If you need to capture the exact handshake failure or verify where packets are dropping, use the pktcap-uw utility on the ESXi host.

  • To capture traffic on the VXLAN interface (vmk10) filtering by the NSX Edge IP:

    pktcap-uw --vmk vmk10 --ip <NSX_EDGE_TEP_IP> -o /tmp/edge_tep_traffic.pcap
  • To capture specifically on port 443 (HTTPS/TLS traffic) to the image proxy:

    pktcap-uw --vmk vmk10 --tcpport 443 -o /tmp/tls_handshake.pcap
    

Press Ctrl+C to stop the capture. You can then export the .pcap files from the /tmp/ directory to analyze in Wireshark.

Step 5: Network Resolution

Engage your physical network team to ensure that all top-of-rack (ToR) switches, core switches, and routers in the path between the ESXi hosts and the NSX Edge nodes are configured to support an MTU of at least 1600 (9000 is recommended).

Once the physical MTU is corrected, the Kubernetes image fetcher will automatically retry, and the pods will transition to a Running state.