TKGI cluster upgrade fails from TKGI 1.14.x to 1.15.x

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:
TKGI upgrade-cluster is failing, no Bosh task is triggering the TKGI cluster upgrade.

Error shown in TKGI pks-api logs:

2023-08-05 00:35:36.038  INFO 26178 --- [      upgrade-0] io.pivotal.pks.upgrade.ClusterUpgrader   : Action 'upgrade-cluster', cluster name: 'mycluster'
2023-08-05 00:36:53.175 ERROR 26178 --- [nio-9021-exec-4] i.pivotal.pks.error.ApiExceptionHandler  : Error: Status: 500; ErrorMessage: <nil>; Description: Expected to find variables: nsxt_node_ip_block_ids; ResponseError: <nil>
2023-08-05 00:36:53.175  WARN 26178 --- [nio-9021-exec-4] i.pivotal.pks.error.ApiExceptionHandler  : No HttpStatus code defined in advice for [class io.pivotal.pks.broker.client.ODBException]
2023-08-05 00:55:36.560 ERROR 26178 --- [      upgrade-0] io.pivotal.pks.upgrade.ClusterUpgrader   : Error while upgrading cluster mycluster: {}
io.pivotal.pks.broker.client.ODBException: Status: 500; ErrorMessage: <nil>; Description: Expected to find variables: nsxt_node_ip_block_ids; ResponseError: <nil>

Cause

The field nsxt_node_ip_block_ids is missing from TKGI cluster manifest that is preventing the TKGI api to trigger the cluster upgrade

Resolution

This issue has been observed in a scenario where a specific TKGI cluster was not fully upgraded before performing a new upgrade on it.

Workaround:
1) Review the manifest for this specific TKGI cluster
You may download the manifest for TKGI cluster using the service-instance_ID and check if the field nsxt_node_ip_block_ids exists
tkgi clusters
bosh -d service-instance_ID manifest > service-instance_ID.yml
grep nsxt_node_ip_block_ids service-instance_ID.yml

2) If the field nsxt_node_ip_block_ids is missing, you may manually add into the manifest file
cp service-instance_ID.yml service-instance_ID_backup.yml
vim service-instance_ID.yml

- Just after the line nsxt_network: true in the manifest, add nsxt_node_ip_block_ids and the name (retrieve it from TKGI tile > settings > Networking > Nodes IP Block ID)

  nsxt_network: true
  nsxt_node_ip_block_ids:
  - name: 1ab1ab11a-1234-410c-be80-786cfd6be1af

3) Deploy the manifest:
bosh -d service-instance_ID deploy service-instance_ID.yml

4) Upgrade the TKGI cluster
tkgi upgrade-cluster CLUSTERNAME