Asymmetric connectivity fails when forwarding logs and metrics from Supervisor cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

   This asymmetry between the two datacenters is due to a built-in networking optimization in vSphere Supervisor called Direct Routing.

   When expose a service on a workload cluster via a LoadBalancer VIP, the NSX Container Plugin (NCP) automatically synchronizes that service information to the Supervisor cluster.

   For clients residing within the same datacenter, Tanzu intentionally configures the local routing to bypass the Load Balancer and send traffic directly to the workload cluster's Node IPs. This is indicated by the service.route.lbapi.run.tanzu.vmware.com/type: direct label on the service object.

Cause

Direct Routing optimization in vSphere Supervisor bypassing Load Balancer for local traffic.

The connectivity failure is caused by a built-in vSphere Supervisor networking optimization known as "Direct Routing." When a LoadBalancer service is exposed in a workload cluster, the VirtualMachineService controller automatically creates a corresponding Service in the local Supervisor cluster with the attribute service.route.lbapi.run.tanzu.vmware.com/type: direct. This configuration instructs the Supervisor's networking layer to bypass the physical Load Balancer for internal traffic and route it directly to the workload nodes to reduce latency.

Resolution

There is currently no supported option to disable this direct routing behavior. The synchronization is managed by the system and any manual overrides would be automatically reverted during cluster state reconciliation.

To ensure reliable connectivity and support granular firewall scoping, you should adjust your DFW strategy to account for Direct Routing:

1. Utilize NSX Security Groups: Create dynamic groups in NSX that automatically include all nodes within the target TKG workload cluster.
2. Dual-Scope Firewall Rules: Configure DFW rules to permit traffic from the Supervisor cluster to both the LoadBalancer VIP (for external/cross-DC consistency) and the Workload Node Security Group (for same-DC direct traffic).
3. Review Article 388260: Refer to KB 388260 for additional packet capture methodologies to validate that traffic is reaching the node interfaces correctly.