Velero Backup Service Fails with net/http: TLS handshake timeout in vSphere Supervisor Environments

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

The Velero backup service within a Supervisor deployment fails to initialize or enters a CrashLoopBackOff state, and logs indicate a failure to communicate with the internal Kubernetes API endpoint.

Symptoms:

• Velero pods restart repeatedly and show a crashloopbackoff status
• Log entries show the following error: An error occurred: Get "https://<INTERNAL_API_IP>:443/api/v1/namespaces/velero": net/http: TLS handshake timeout
• Running velero version only returns the Client version, while the Server version remains blank or reports a connection error.

Environment

• VMware Cloud Foundation (VCF)
• vSphere Supervisor Cluster / Guest Clusters
• VMware vSphere Kubernetes Service (vK8s) 3.3.3 and earlier
• Velero Plugin for vSphere

Cause

This issue occurs due to a version mismatch between the Velero client/plugin and the vSphere Supervisor version.
When an incompatible Velero client is used against an older Supervisor deployment that requires specific plugin versions (e.g., v1.7.x or earlier), the TLS handshake cannot be completed because the client cannot properly negotiate API requests with the internal Kubernetes API server.

Resolution

Install the Velero client and plugin versions with the requirements of your specific Supervisor build.

1. Prerequisites:

   • Verify your current vCenter and Supervisor build versions.
   • Ensure you have administrative access to the Supervisor cluster.

2. Uninstall Incompatible Components: Remove the currently installed incompatible Velero client from from the Supervisor. Uninstalling Velero

3. Download and Install Compatible Version: Download the Velero client version that is explicitly supported for your Supervisor build (typically v1.7.x or earlier for older deployments). Refer to the product interoperability matrix for exact version mapping.

4. Install the Plugin: Use the compatible client to install the plugin to the Supervisor following the official configuration guides: 
   Backup and Restore vSphere Pods Using the Velero Plugin for vSphere
   Install and Configure the Velero Plugin for vSphere Version 1.7.x and Earlier on a Supervisor

5. Verify Connectivity:
   Run the following command within the supervisor and validate both values return successfully

   velero version

   Validate all Velero pods are working and ready after the re-deployment.

Additional Information

For issues persisting after version alignment, verify Path MTU (PMTU) and ensure Jumbo Frames (MTU 9000) are configured end-to-end across the physical and virtual network infrastructure.