Resizing Tanzu Kubernetes Grid Integrated Edition clusters fails with the erorr, "No config: error fetching latest cloud config"

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Symptoms:

• When resizing a Tanzu Kubernetes Grid Integrated Edition (TKGI) cluster, you see a message similar to the following. 

Update summary for cluster my-cluster:
Worker Number: 6
Are you sure you want to continue? (y/n): y
Error: Error processing update parameters: error fetching latest cloud config with name pks-service-instance_9d9a8b55-1fef-4c51-846e-687eaa5e260a, error No config: error fetching latest cloud config with name service-instance_service-instance_9d9a8b55-1fef-4c51-846e-687eaa5e260a: No config

 

• You see messages noting the UUID of the cluster similar to the following in the pks-nsx-t-osb-proxy.stdout.log file:

{"timestamp":"1620083297.006016254","source":"pks-nsx-t-osb-proxy","message":"pks-nsx-t-osb-proxy.bosh.DNSConfig error","log_level":1,"data":{"cloud-config-name":"service-instance_9d9a8b55-1fef-4c51-846e-687eaa5e260a","error":"No config","instance-id":"9d9a8b55-1fef-4c51-846e-687eaa5e260a","s
ession":"1"}}
{"timestamp":"1620083297.012918949","source":"pks-nsx-t-osb-proxy","message":"pks-nsx-t-osb-proxy.bosh.DNSConfig error","log_level":1,"data":{"cloud-config-name":"pks-9d9a8b55-1fef-4c51-846e-687eaa5e260a","error":"No config","instance-id":"9d9a8b55-1fef-4c51-846e-687eaa5e260a","session":"1"}}
{"timestamp":"1620083297.013795614","source":"pks-nsx-t-osb-proxy","message":"pks-nsx-t-osb-proxy.unknown-error","log_level":2,"data":{"error":"Error processing update parameters: error fetching latest cloud config with name pks-9d9a8b55-1fef-4c51-846e-687eaa5e260a, error No config: error fetching latest cloud config with name service-instance_9d9a8b55-1fef-4c51-846e-687eaa5e260a: No config"}}

 

• You see output similar to the following in the from the bosh configs command (the "Team" value is missing for one or more of the service-instances). Note: Only "pivotal-container-service-###" and "service-instance_###" should belong to the "pivotal-container-service-###" Team. Others may or may not belong to a team depending on the tiles deployed. 

Using environment '172.##.#.#' as client 'ops_manager'

ID   Type     Name                                                                    Team                                            Created At
6*   cloud    default                                                                 -                                               2021-02-18 14:53:48 UTC
7*   cloud    pivotal-container-service-55ea8c93695d5adad0b2                          pivotal-container-service-55ea8c93695d5adad0b2  2021-02-18 15:49:45 UTC
14*  cloud    service-instance_9d9a8b55-1fef-4c51-846e-687eaa5e260a                   -                                               2021-02-19 22:48:18 UTC
5*   cpi      default                                                                 -                                               2021-02-18 14:02:02 UTC
3*   runtime  director_runtime                                                        -                                               2021-02-18 14:02:00 UTC
8*   runtime  harbor-container-registry-97e60c212907d2fd5a61-harbor-bosh-dns-aliases  -                                               2021-02-18 16:00:53 UTC
1*   runtime  ops_manager_dns_runtime                                                 -                                               2021-02-18 14:01:58 UTC
2*   runtime  ops_manager_system_metrics_runtime                                      -                                               2021-02-18 14:01:59 UTC

(*) Currently active
Only showing active configs. To see older versions use the --recent=10 option.

8 configs

Succeeded

Cause

If a different user updates the config, the Team can be lost (for example, if using the BOSH cli bosh_client instead of the TKGI bosh_client). This prevents the TKGI client from accessing the config as it no longer has ownership.

Resolution

This is a known issue affecting Tanzu Kubernetes Grid Integrated Edition. There is currently no resolution.

Workaround:
To workaround this issue,

1. Get the current config by issuing a command similar to the following:

bosh config --column=content <id> > service-instance_uuid-id.yml
 

Note: If there is a blank line at the end of the service-instance_uuid-id.yml file, it must be removed.

2. Since it is often the BOSH admin account that made the changes that dropped the Team it will need to be deleted first before the TKGI bosh_client can change/recreate it. Issue a command similar to the following:

bosh delete-config --type=cloud --name=service-instance_uuid

3. Setup the TKGI user environment "BOSH_CLIENT" and "BOSH_CLIENT_SECRET" details from the TKGI tile. The credentials can be found from the PKS Tile>Credentials>uaa_client_credentials page in the Opsman UI.

export BOSH_CLIENT="pivotal-container-service-####"
export BOSH_CLIENT_SECRET="###"

4. Update the bosh config using the TKGI bosh_client by issuing a command similar to the following:

bosh update-config --type=cloud --name=service-instance_uuid service-instance_uuid-id.yml

5. You can attempt to resize the cluster again at this point.

Note: If you see a message similar to "Director responded with non-successful status code '401' response '{"code":600000,"description":"Require one of the scopes: bosh.admin, bosh.62cc6301-ab70-4e6c-b7cb-474c2731e039.admin" it is due to the config not being deleted before switching to the TKGI bosh_client. Change back to the bosh tile cli credentials and delete the config first before trying the update at the TKGI bosh_client.