The svc-auto-attach package fails to reconcile. You may observe the following:
The auto-attach pod is stuck in an ErrImagePull or CrashLoopBackOff state.
Checking the Image CR or pod events shows the following error:
Failed to resolve on node <hostname>. Reason: Http request failed. Code 400: ErrorType(2) failed to do request: Head "https://mgmt-image-proxy.kube-system.svc.cluster.local/...": net/http: TLS handshake timeout
VMware vSphere Kubernetes Service
VMware Cloud Foundry 9.x
This issue can be caused by various reasons which could interfere end-end IP communications.
- MTU Mismatch
- No route between endpoint
- Misconfiguration in underlying Networking layer
- Firewall Settings Blocking the communication
For example, in this specific case it is caused by an MTU mismatch in the physical network or on the ESXi VMkernel ports.
Overlay networking (VXLAN/Geneve) requires Jumbo Frames (MTU 1600+). While small packets might pass successfully, large packets (like those generated during a TLS certificate exchange for an image pull) will be fragmented or dropped if any physical switch or VMkernel adapter in the path is restricted to a standard MTU of 1500. This results in the TLS handshake timeout.
Before isolating the ESXi hosts, rule out Control Plane or Firewall issues:
Check Pod Events: Run kubectl describe pod <pod-name> -n <namespace> to look for CNI errors or confirm the TLS timeout.
Check NCP Logs: Review the vmware-system-netop namespace logs for IPAM exhaustion (e.g., "Insufficient IP capacity in the subnet").
NSX Traceflow / DFW: Use the NSX Traceflow UI to ensure a Distributed Firewall "Default Drop" rule isn't blocking the traffic before blaming MTU.
Ensure the ESXi interfaces handling overlay traffic (typically vmk10 and vmk11) and management (vmk0) are configured for Jumbo Frames (9000). Run the following on the ESXi host:
esxcfg-vmknic -l
Check the MTU column to ensure the appropriate interfaces are set to 9000 rather than 1500.
Identify your NSX Edge TEP IPs and run a ping test from the ESXi host's VXLAN VMkernel port to verify if large packets are being dropped in the physical network.
Note: Standard tools like nc -vz or curl without large payloads might succeed because small packets do not exceed the 1500 MTU limit. A TLS handshake exchanges certificates, creating large packets that exceed the standard MTU once overlay encapsulation (VXLAN/Geneve) is added. This is why testing with the -s 1572 and -d (Do Not Fragment) flags is mandatory.
Test with Standard MTU (1472 payload + 28 byte header = 1500 MTU):
vmkping -S vxlan -s 1472 -I vmk10 -d <NSX_EDGE_TEP_IP>
(If this succeeds, basic routing is working).
Test with Jumbo Frames (1572 payload + 28 byte header = 1600 MTU):
vmkping -S vxlan -s 1572 -I vmk10 -d <NSX_EDGE_TEP_IP>
(If this fails with 100% packet loss, an upstream physical switch is dropping the jumbo frames).
If you need to capture the exact handshake failure or verify where packets are dropping, use the pktcap-uw utility on the ESXi host.
To capture traffic on the VXLAN interface (vmk10) filtering by the NSX Edge IP:
To capture specifically on port 443 (HTTPS/TLS traffic) to the image proxy:
pktcap-uw --vmk vmk10 --tcpport 443 -o /tmp/tls_handshake.pcap
Press Ctrl+C to stop the capture. You can then export the .pcap files from the /tmp/ directory to analyze in Wireshark.
Engage your physical network team to ensure that all top-of-rack (ToR) switches, core switches, and routers in the path between the ESXi hosts and the NSX Edge nodes are configured to support an MTU of at least 1600 (9000 is recommended).
Once the physical MTU is corrected, the Kubernetes image fetcher will automatically retry, and the pods will transition to a Running state.