TKG management cluster creation on AWS fails with VpcReconciliationFailed

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:
Creating a TKG management cluster on AWS fails with the following error

To clean up the resources created by the management cluster:
tkg delete mc oom-tkgm
unable to set up management cluster,: unable to wait for cluster and get the cluster kubeconfig: error waiting for cluster to be provisioned (this may take a few minutes): cluster creation failed, reason:'VpcReconciliationFailed', message:'0 of 8 completed'

Environment

VMware Tanzu Kubernetes Grid 1.x

Cause

To find the root cause of these types of errors,  capa-controller manager logs are a good place to start

These logs can be obtained by using the following command

kubectl logs deployment.apps/capa-controller-manager -n capa-system manager > capa-system-deployment.log

From the logs, it is clear that this is an issue where the AWS account has reached the maximum number of VPC it can create. 

I1219 19:50:15.219038 1 awsmachine_controller.go:457] controllers/AWSMachine "msg"="Cluster infrastructure is not ready yet" "awsMachine"="oom-tkgm-md-0-84mlb" "cluster"="oom-tkgm" "machine"="oom-tkgm-md-0-6d8bd87fbd-zcg8q" "namespace"="tkg-system" E1219 19:50:15.314176 1 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to reconcile network for AWSCluster tkg-system/oom-tkgm: failed to create new vpc: failed to create vpc: VpcLimitExceeded: The maximum number of VPCs has been reached.\n\tstatus code: 400, request id: 92f56f54-1b1e-4fa3-b0f5-d56be956fe30" "controller"="awscluster" "name"="oom-tkgm" "namespace"="tkg-system" I1219 19:50:15.315595 1 awscluster_controller.go:160] controllers/AWSCluster "msg"="Reconciling AWSCluster" "awsCluster"="oom-tkgm" "cluster"="oom-tkgm" "namespace"="tkg-system"

Resolution

The default quota for the number of VPCs per region is 5. Read more about the latest VPC quotas here.