The service VIP might be allocated to the wrong network when TKG clusters with `AVI_CONTROL_PLANE_HA_PROVIDER:​​ true` upgrade from 1.6 to 2.1

Products

• Tanzu Kubernetes Grid

Issue/Introduction

When AVI_CONTROL_PLANE_HA_PROVIDER is set to true, and AVI_MANAGEMENT_CLUSTER_VIP_NETWORK_NAME, AVI_MANAGEMENT_CLUSTER_CONTROL_PLANE_VIP_NETWORK_NAME, set to different networks, upgrading the cluster might cause AKO to give error like below:

2024-06-20T09:59:39.231Z [33mWARN[0m rest/rest_operation.go:273 RestOp method PUT path /api/vsvip/vsvip-######-9dc4-4a7b-80ee-########### tenant admin Obj {"_last_modified":"171887#######6780","cloud_ref":"https://#####.#####.lab/api/cloud/cloud-e1b94bbd-6a3e-46b9-####-26d3640736b2#tanzu-vcenter01","east_west_placement":false,"markers":[{"key":"clustername","values":["tkg-system-tkgm-mgmt"]}],"name":"tkg-system-tkgm-mgmt--tkg-system-tkg-system-tkgm-mgmt-control-plane","tenant_ref":"https://###.#####.####/api/tenant/admin#admin","url":"https://####.#####.lab/api/vsvip/vsvip-ce03549c-#####-######-####-d13d33c49a03#tkg-system-tkgm-mgmt--tkg-system-tkg-system-tkgm-mgmt-control-plane","uuid":"vsvip-ce03549c-####-4a7b-####-#######","vip":[{"auto_allocate_ip":true,"ipam_network_subnet":{"network_ref":"/api/network/?name=tkg-mgmt-vip","subnet":{"ip_addr":{"addr":"###.16.##.0","type":"V4"},"mask":24}},"vip_id":"0"}],"vrf_context_ref":"https://#####.####.lab/api/vrfcontext/vrfcontext-#######-668b-######-b803-8913113d4c1e#global","vsvip_cloud_config_cksum":"133####64"} returned err {"code":0,"message":"map[error:[tkg-system-tkgm-mgmt--tkg-system-tkg-system-tkgm-mgmt-control-plane#0] No suitable network found, possible reasons:\n1.Check if networks of interest are added to the usable_networks in ipam_profile.2.Check if ipam_network_subnet in the vip has valid network and subnet(s) configured.3.Check if the networks have configured_subnets with static pool of IPs.]","Verb":"PUT","Url":"https://####.#####.lab//api/vsvip/vsvip-ce03549c-9dc4-####-80ee-############","HttpStatusCode":400} with response null

Check if networks of interest are added to the usable_networks in ipam_profile.2.Check if ipam_network_subnet in the vip has valid network and subnet(s) configured.3.Check if the networks have configured_subnets with static pool of IPs

Cause

This is because of a bug in AKOO 1.6, that it doesn’t wait for AVIInfrasetting to be ready first when creating the service for the control plane endpoint. So the service might not have the correct AVIInfrasetting annotation like below

For Management cluster, it’s missing below annotation

aviinfrasetting.ako.vmware.com/name: install-ako-for-management-cluster-ais

For workload cluster, it’s missing below annotation

aviinfrasetting.ako.vmware.com/name: install-ako-for-all-ais

This is fixed in AKOO 1.8, which is in TKG 2.2

Resolution

New cluster in TKG 2.2 won’t have this issue.

 

Workaround:

For Management cluster, if it’s missing below annotation on the service for control plane HA, for example ‘tkg-system-tkgm-mgmt-control-plane’, add below annotation:

aviinfrasetting.ako.vmware.com/name: install-ako-for-management-cluster-ais

For workload cluster, it’s missing below annotation on the service for control plane HA, for example ‘default-test-cluster-control-plane’, add below annotation:

aviinfrasetting.ako.vmware.com/name: install-ako-for-all-ais