TKG : Orphaned route tables not removed when upgrading Kubernetes versions in Azure

Products

• Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:
kubernetes-controller-manager logs report the following error:
 

Warning FailedToCreateRoute 52s (x9070 over 15d) route_controller (combined from similar events): Could not create route 12345678-xxxx-xxxx-xxxx-6ab6a147598d 100.96.4.0/24 for node dev-control-plane-v1-21-2-vmware-1-6kh7s-xkv78 after 30.137845507s: Retriable: false, RetryAfter: 0s,
HTTPStatusCode: 400, RawError: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: {"error": {"code": "RouteConflict", "message": "Route: dev-control-plane-v1-21-2-vmware-1-6kh7s-xkv78__100964024 conflicts with Route: dev-control-plane-jl7c8. Routes cannot have the same AddressPrefix.", "details": []}

Cause

The in-tree Azure cloud provider creates routes and route tables that persist through Kubernetes upgrades and are unable to be deleted or updated when the pod CIDR of the new control plane matches that of the orphaned route.

Resolution

The use of the route controller is not necessary when using TKG in Azure environments. The configure-cloud-routes flag will be set to false in future releases of TKG

Workaround:

Disable the route controller by adding the flag configure-cloud-routes: "false" to the controller manager extra args in the KubeadmControlPlane similar to the example below

 

spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      controllerManager:
        extraArgs:
          configure-cloud-routes: "false"

Additional Information

This may affect all TKG versions.