PKS with NSXT configured does not update the node DNS on existing clusters when updated on the tile.

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Issue:

PKS with NSXT configured does not update the node DNS on existing clusters when updated on the tile.

Tested operations:

 Tried updating the secondary DNS inside PKS tile NODE_DNS, applied the changes from opsman selecting upgrade all clusters errand, it did not change for existing clusters, tried pks upgrade-cluster, tried enabling “Recreate all VMs” in BOSH tile and then applied the changes selecting upgrade all clusters errand and also did a bosh deploy -d service-instance manifest.yml --recreate all were successful but the DNS servers are not added

Environment

Product Version: 1.10

Resolution

Workaround:

Very Important first step;  because the pks broker is not a bosh admin and so cannot see the update config to bosh login as the pks user and and so cannot see the update config, the BOSH CLI operations have to be completed with 
"BOSH_CLIENT" and "BOSH_CLIENT_SECRET" environment details get the credentials from the PKS Tile/Credentials/uaa_client_credentials
For example: 
export BOSH_CLIENT=pivotal-container-service-cedfe8e.....cadcd BOSH_CLIENT_SECRET=135a9......83458bbf BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate BOSH_ENVIRONMENT=<DIRECTOR_IP>

 

Identify the instance ID from pks cli

pks clusters in the response take record of the ID for the target cluster <ID>

 

 

bosh configs | grep <ID>

get the ID for the config (first column) <configID>

 

bosh config <configID>

confirm the correct <ID> is specified in the output

 

Export the output:

bosh config <configID> > config.yml

 

Edit the file:

 

Origin:

ID 76

Type cloud

Name service-instance_3f05330a-43a5-41ae-ad79-a6c16a931ccb

Created At 2020-07-10 09:36:07 UTC

Content networks:

- name: pks-3f05330a-43a5-41ae-ad79-a6c16a931ccb

subnets:

- azs:

- PKS-MGMT-1

cloud_properties:

name: pks-3f05330a-43a5-41ae-ad79-a6c16a931ccb

dns:

- 10.192.2.10

- 10.192.2.11

gateway: 172.34.16.1

range: 172.34.16.0/24

type: manual

vm_extensions:

- cloud_properties: {}

name: service-instance-3f05330a-43a5-41ae-ad79-a6c16a931ccb-master-nsgroup

           

Modify the file by removing the top 4 lines and the last empty line and add necessary DNS records:

 

Updated

 

Content networks:

- name: pks-3f05330a-43a5-41ae-ad79-a6c16a931ccb

subnets:

- azs:

- PKS-MGMT-1

cloud_properties:

name: pks-3f05330a-43a5-41ae-ad79-a6c16a931ccb

dns:

- 10.192.2.10

- 10.192.2.11

- 8.8.8.8

gateway: 172.34.16.1

range: 172.34.16.0/24

type: manual

vm_extensions:

- cloud_properties: {}

name: service-instance-3f05330a-43a5-41ae-ad79-a6c16a931ccb-master-nsgroup

 

once file is updated use bosh update-config to update the existing config:

bosh update-config --type=cloud --name=service-instance_<ID> config.yml

 

Verify and Confirm the update
To confirm updated values:

 

bosh configs | grep <ID>

get the ID for the config (first column) <configID>

 

bosh config <configID>

confirm the correct <ID> is specified in the output

 

 

Upgrade the cluster with

pks upgrade-cluster <clustername>

 

The cluster VMs will be recreated with the new DNS in place to confirm login into any of the newly created VMs from the deployment and verify /etc/resolv.conf for the new DNS entrance

 

 

cat /etc/resolv.conf

# This file was automatically updated by bosh-dns

nameserver 169.254.0.2

 

nameserver 10.192.2.10

nameserver 10.192.2.11

nameserver 8.8.8.8