Tanzu Kubernetes Grid management cluster creation fails with reason:'BastionHostFailed', message:'6 of 8 completed'

Products

• Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:

• You are attempting to create a TKG management cluster in AWS and it fails during the creation of bastion host
• You observe the below error returned by the cli

unable to set up management cluster: unable to wait for cluster and get the cluster kubeconfig: error waiting for cluster to be provisioned (this may take a few minutes): cluster creation failed, reason:'BastionHostFailed', message:'6 of 8 completed'

Environment

VMware Tanzu Kubernetes Grid 1.x

Cause

To find the root cause of this problem, you should look at the capa-controller-manager pod logs in the capa-system namespace:

kubectl logs deployment.apps/capa-controller-manager -n capa-system manager

If you see the below error in your logs, the error occurs because the Bastion host AMI lookup fails in the region where you are creating your cluster. 

E1221 23:21:57.608603 1 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to reconcile bastion host for AWSCluster tkg-system/oom: failed to run instance: InvalidAMIID.Malformed: Invalid id: \"unknown region\" (expecting \"ami-...\")\n\tstatus code: 400, request id: #####" "controller"="awscluster" "name"="oom" "namespace"="tkg-system"

You will observe this error if you are deploying a cluster in a region where bastion host AMI is not available. us-gov-west is an example of such a region.

Resolution

You must switch to one of the regions where bastion host AMI is available. You can get an exhaustive list of regions where AMI lookup is done at cluster-api-provider-aws/pkg/cloud/services/ec2/ami.go  Please refer to function defaultBastionAMILookup for the version of cluster-api-provider-aws shipped with your TKG version.

An alternate option is to disable bastion host creation. This can be done from the TKG installer UI and also by setting  BASTION_HOST_ENABLED: "false" in the TKG configuration.