PKS / TKGI with NSX-T configured does not update the node DNS on existing clusters when updated on the tile

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Issue

PKS (now VMware Tanzu Kubernetes Grid Integrated Edition or TKGI) with NSX-T configured does not update the node DNS on existing clusters when updated on the tile.

What actions can the user take to resolve or work around the described problem? This article will describe a workaround for the issue.

Operations that were tested

• Tried updating the secondary DNS inside the TKGI tile, NODE_DNS, and applied the changes from OpsMan.
• Selected the upgrade all clusters errand and it did not change for existing clusters.
• Tried pks upgrade-cluster.
• Tried enabling “Recreate all VMs” in the BOSH tile and then applied the changes, selecting the upgrade all clusters errand
• Tried a bosh deploy -d service-instance manifest.yml --recreate and all were successful but the DNS servers are not added.

Environment

Product Version: 1.10

Resolution

Resolution


From TKGi 1.11, operator can add DNS server IP to TKGi cluster using network profile as per https://docs.pivotal.io/tkgi/1-11/release-notes.html#1-11-0-features section "Add DNS Server IPs to Clusters Using Network Profiles"



Workaround

Identify the instance ID from pks cli. TKGI clusters in the response take record of the ID for the target cluster <ID>.
 

Important Step

The TKGI broker is not a bosh admin, so it cannot see the update config to bosh login as the TKGI user.

The bosh cli operations have to be completed with the "BOSH_CLIENT" and "BOSH_CLIENT_SECRET." Get the credentials from PKS Tile / Credentials / uaa_client_credentials.

For example: 

export BOSH_CLIENT=pivotal-container-service-cedfe8e.....cadcd BOSH_CLIENT_SECRET=135a9......83458bbf BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate BOSH_ENVIRONMENT=<DIRECTOR_IP>

bosh configs | grep <ID>

Get the ID for the config (first column) <configID>:

bosh config <configID>

Confirm the correct <ID> is specified in the output.
 
Export the output:

bosh config <configID> > config.yml

 
Edit the file:

Origin:
ID 76
Type cloud
Name service-instance_3f05330a-43a5-41ae-ad79-a6c16a931ccb
Created At 2020-07-10 09:36:07 UTC
Content networks:
- name: pks-3f05330a-43a5-41ae-ad79-a6c16a931ccb
subnets:
- azs:
- PKS-MGMT-1
cloud_properties:
name: pks-3f05330a-43a5-41ae-ad79-a6c16a931ccb
dns:
- 10.192.2.10
- 10.192.2.11
gateway: 172.34.16.1
range: 172.34.16.0/24
type: manual
vm_extensions:
- cloud_properties: {}
name: service-instance-3f05330a-43a5-41ae-ad79-a6c16a931ccb-master-nsgroup         

Modify the file by removing the top 4 lines and the last empty line and add the necessary DNS records. The updated file is pasted below:

Content networks:
- name: pks-3f05330a-43a5-41ae-ad79-a6c16a931ccb
subnets:
- azs:
- PKS-MGMT-1
cloud_properties:
name: pks-3f05330a-43a5-41ae-ad79-a6c16a931ccb
dns:
- 10.192.2.10
- 10.192.2.11
- 8.8.8.8
gateway: 172.34.16.1
range: 172.34.16.0/24
type: manual
vm_extensions:
- cloud_properties: {}
name: service-instance-3f05330a-43a5-41ae-ad79-a6c16a931ccb-master-nsgroup

Once the file is updated, use bosh update-config to update the existing config:

bosh update-config --type=cloud --name=service-instance_<ID> config.yml

 
Now, verify and confirm the update. To confirm the updated values, follow the instructions below:

bosh configs | grep <ID>

Get the ID for the config (first column) <configID>.

bosh config <configID>

Confirm the correct <ID> is specified in the output.
 
Upgrade the cluster with the following:

pks upgrade-cluster <clustername>

 
The cluster VMs will be recreated with the new DNS in place to confirm login into any of the newly created VMs from the deployment. It will also verify /etc/resolv.conf for the new DNS entrance:

cat /etc/resolv.conf
# This file was automatically updated by bosh-dns
nameserver 169.254.0.2
 
nameserver 10.192.2.10
nameserver 10.192.2.11
nameserver 8.8.8.8