Missing links in the BOSH database cause 'bosh recreate' and 'bosh cloud-check' to fail in PKS

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Symptoms:
Attempting to recreate a cluster VM or the cluster deployment fails when links are missing in the BOSH database.  Running the commands, bosh recreate or bosh cloud-check, fails with errors similar to this:

- Unable to render templates for job 'smoke-tests'. Errors are: - Error filling in template 'ca.pem.erb' (line 1: Can't find link 'kube-apiserver') - Error filling in template 'kubeconfig.erb' (line 1: Can't find link 'kube-apiserver')

Environment

Cause

The above error indicates that there are missing records in the 'links' table of the BOSH database.

Resolution

Since bosh recreate or bosh cloud-check of the deployment fails with the same error, run `pks resize` or `pks update-cluster` on the impacted cluster and BOSH will redeploy the cluster. Use the same number of nodes to ensure there will be no changes to the cluster:

For example:

# to redeploy a cluster that currently has 2 workers
$ pks resize <cluster-name> -n 2

# `pks resize` will be deprecated soon.  In case resize is no longer there, use `pks update-cluster` to redeploy a cluster that currently has 2 workers
$ pks update-cluster <cluster-name> --num-nodes 2