PKS create-cluster failing in apply-addons errand on GCP

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Symptoms:
When running a new PKS foundation running on GCP, cluster creation. through `pks create-cluster`, eventually fails in the apply-addons stage. 

It fails because CoredDNS cannot be deployed successfully. In other words, the kube-controller-manager cannot communicate with the GCP API. Since the cluster's master and worker VM's are not assigned External IP addresses, they are not able to reach out to external networks.

The BOSH logs of the task show the following:

{"instance":{"group":"apply-addons","id":"xxxxxxxx-71dd-yyyy-9a8e-4f23d0x1xxxx"},"errand_name":"apply-addons","exit_code":1,"stdout":"Deploying /var/vcap/jobs/apply-specs/specs/coredns.yml\nserviceaccount/coredns created\nclusterrole.rbac.authorization.k8s.io/system:coredns created\nclusterrolebinding.rbac.authorization.k8s.io/system:coredns created\nconfigmap/coredns created\ndeployment.extensions/coredns created\nservice/kube-dns created\nWaiting for deployment spec update to be observed...\nfailed to start all system specs after 1200 with exit code 124\n","stderr":"","logs":{"blobstore_id":"xxxxxxxx-fdd9-yyyy-6bee-6457b4xd37f4","sha1":"xxxx030c283xxxx642dc62631ddeaa60ab15xxxx"}}

Note: Another indication on an issue is that there is no Network Address Translation (NAT) instance or VM in GCP for the services network.

Environment

Cause

The terraform templates for PKS (v0.74.0) do not include a NAT instance for the services network used by the cluster VM's.

Resolution

The workaround is to create a NAT instance for the services subnet.

Step 4 in "Preparing to Deploy Ops Manager on GCP Manually" can be followed to create the NAT instance manually.

After the NAT instance has been created, delete the failed cluster and then create a new cluster. The new cluster creation should complete successfully.