Errors while configuring Supervisory Cluster | Fails Due to Asymmetric Routing Between vCenter and Supervisor Control Plane VMs

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Errors while configuring the Supervisor Cluster.

During Supervisor enablement, the operation fails shortly after the Control Plane VMs (CPVMs) are deployed.

In wcpsvc.log, repeated timeouts appear when vCenter attempts to reach the Supervisor floating IP.

The Control Plane VM kernel logs contain messages similar to the following:

IPv4: martian source <CPVM_FIP> from <VCSA_IP>, on dev eth0

Example topology:

• VCSA: 192.168.50.10/24
• CPVM eth0 (management): 10.10.10.5/27 with FIP 10.10.10.4
• CPVM eth1 (workload): 192.168.50.11/24

Supervisor enablement stalls because vCenter cannot communicate with the Supervisor API endpoint (FIP).

The following error is seen in the vCenter UI under Workload Management:

Installed and Started Kubernetes Node Agent on the ESXi Host A general system error occurred. Error message: Get "http://localhost:1080/external-cert/http1/<vip>/6443/api/v1/nodes?fieldSelector=metadata.name%3D<host>": context deadline exceeded (Client. Timeout exceeded while awaiting headers)

 

While SSH directly to the affected Supervisor control plane VMs, the following command returns a networking error reaching the vCenter:

/usr/lib/vmware-wcp/wcp-sv-state-checker -v debug -c ConnectToVCenter

 

This issue can also occur during a Supervisor cluster upgrade where Workload Management/Supervisor Management shows the error:

The System VM for solution <Supervisor VM DNS name> on cluster domain-c## is expected to be powered-on, but it is powered-off.

Cause

The issue occurs when the Supervisor Control Plane VMs are dual-homed and vCenter resides on the same subnet as the CPVM workload interface instead of the management interface.

This creates asymmetric routing, traffic from vCenter to the Supervisor FIP (on the management network) returns through the CPVM’s workload interface.

Because asymmetric routing is disabled on Control Plane VMs, the kernel drops these return packets and logs them as martian sources.

As a result, vCenter cannot complete the Supervisor enablement.

Resolution

Resolution:

Ensure symmetric routing between vCenter and the Supervisor Control Plane VMs:

1. Option 1 – Move the vCenter
   • Place vCenter on the same subnet as the Supervisor management interface (for example, 10.10.10.0/27).
     
     
2. Option 2 - Check networking reverse-path-filter settings
   

   • A potential workaround would be to configure reverse-path-filter to a mode that would allow for asymmetrical routing to function consistently. However, this is not a permanent fix because it will be easily reverted.

3. Option 3 – Redeploy the Supervisor cluster
   • Deploy the Supervisor cluster so that its management network resides on the same subnet as vCenter (for example, 192.168.50.0/24), while the workload interface uses a different VLAN or subnet.
     
     
4. After correcting the network alignment, disable and re-enable Supervisor to redeploy the Control Plane VMs.
   
   
5. Verify successful communication by confirming that wcpsvc.log no longer shows timeouts and that no new martian source entries appear in the logs.

 

Note: Asymmetric routing is not supported for Supervisor management communication. All traffic between vCenter and the Supervisor control plane must flow through the same network path.