Attempts to resolve hostnames that have a domain suffix that ends in ".local" fail on Tanzu Kubernetes Grid (TKG) nodes.
VMware Telco Cloud Automation 2.3
A known issue is documented in Tanzu Kubernetes Grid (TKG) as per VMware KB: Search cancel Search Tanzu Kubernetes Grid nodes are unable to resolve hostnames with the ".local" domain suffix
This is a known issue affecting Tanzu Kubernetes Grid. There is currently no resolution in TCA 2.3/TCA3.0.
Workaround:
To deploy TKG Management Cluster with .local domain from TCA 2.3/ 3.0 UI:
Step 1:
A. For TCA 2.3 - SSH to TCA-CP with Admin credentials > Switch to root user > and then proceed with Step C:
B. For TCA 3.0 SSH to TCA-CP with Admin credentials and perform the following commands then proceed with step C:
admin@tca [ ~ ]$ kubectl get pods -A|grep tkg230 tca-cp-cn kbs-tkg230-55795ccb89-jlrdz 1/1 Running 2 (8d ago) 28d admin@tca [ ~ ]$ kubectl exec -n tca-cp-cn kbs-tkg230-55795ccb89-jlrdz -it bash kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. Defaulted container "kbs-tkg230-operator" out of: kbs-tkg230-operator, init-bootstrapper (init)
C. Update the file “config_default.yaml” for the below parameters with the correct .local domain under the directory “/root/.config/tanzu/tkg/providers/”.
#! Specifies the service domain for the cluster SERVICE_DOMAIN: #! Node SearchDomains CONTROL_PLANE_NODE_SEARCH_DOMAINS: vmware.local WORKER_NODE_SEARCH_DOMAINS: vmware.local
D. Create below two files (vsphere-overlay-dns-control-plane.yaml | vsphere-overlay-dns-workers.yaml) with the content in the directory “/root/.config/tanzu/tkg/providers/infrastructure-vsphere/ytt/”
vi vsphere-overlay-dns-control-plane.yaml
#@ load("@ytt:overlay", "overlay") #@ load("@ytt:data", "data") #@overlay/match by=overlay.subset({"kind":"VSphereMachineTemplate", "metadata": {"name": data.values.CLUSTER_NAME+"-control-plane"}}) --- spec: template: spec: network: devices: #@overlay/match by=overlay.all, expects="1+" - #@overlay/match missing_ok=True nameservers: ["172.22.22.43"] #@overlay/match missing_ok=True searchDomains: ["vmware.local"]
vi vsphere-overlay-dns-workers.yaml
#@ load("@ytt:overlay", "overlay") #@ load("@ytt:data", "data") #@overlay/match by=overlay.subset({"kind":"VSphereMachineTemplate", "metadata": {"name": data.values.CLUSTER_NAME+"-worker"}}) --- spec: template: spec: network: devices: #@overlay/match by=overlay.all, expects="1+" - #@overlay/match missing_ok=True nameservers: ["172.22.22.43"] #@overlay/match missing_ok=True searchDomains: ["vmware.local"]
Step2: Deploy the TKG Management cluster from TCA UI.
Step 3: After TKG Management cluster creation execute the below steps before starting v2 TKG Workload Cluster.
3a: SSH to the TKG Management cluster with CAPV user
3b: Identify the "tca-kubecluster-operator" pod:
capv@tcpran30-mgmt-9fqh5-k8fkt [ ~ ]$ kubectl get pods -n tca-system NAME READY STATUS RESTARTS AGE nodeconfig-daemon-97csj 1/1 Running 0 121m nodeconfig-daemon-control-plane-4lhtw 1/1 Running 0 121m nodeconfig-daemon-control-plane-h9s5c 1/1 Running 0 121m nodeconfig-daemon-control-plane-s8w6b 1/1 Running 0 121m nodeconfig-daemon-fcjpj 1/1 Running 0 121m nodeconfig-operator-7fcd9b44b8-qtvff 2/2 Running 0 121m tca-diagnosis-operator-548d99df66-8276c 1/1 Running 0 120m tca-kubecluster-operator-6879b6bd48-fv697 1/1 Running 0 120m test-controller-65d68cf67-9lqbn 1/1 Running 0 121m vmconfig-operator-bfd9c589-2qrf7 2/2 Running 2 (119m ago) 119m
3c: Create the above same files with same content in the management cluster master node and copy it to the tca-kubecluster-operator-6879b6bd48-fv697 under “/root/.config/tanzu/tkg/providers/infrastructure-vsphere/ytt/” as mentioned in the above Step 1-D.
Example:
capv@tcpran30-mgmt-9fqh5-k8fkt [ ~ ]$ kubectl -n tca-system cp vsphere-overlay-dns-control-plane.yaml tca-kubecluster-operator-6879b6bd48-fv697:/root/.config/tanzu/tkg/providers/infrastructure-vsphere/ytt/
capv@tcpran30-mgmt-9fqh5-k8fkt [ ~ ]$ kubectl -n tca-system cp vsphere-overlay-dns-workers.yaml tca-kubecluster-operator-6879b6bd48-fv697:/root/.config/tanzu/tkg/providers/infrastructure-vsphere/ytt/
3d: Login to the POD and validate the files:
capv@tcpran30-mgmt-9fqh5-k8fkt [ ~ ]$ kubectl exec tca-kubecluster-operator-6879b6bd48-fv697 -n tca-system -it bash NOTE: kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. root [ / ]# ls -al /root/.config/tanzu/tkg/providers/infrastructure-vsphere/ytt total 32 drwxr-xr-x 1 root root 4096 2023-05-08 10:01 . drwxr-xr-x 1 root root 4096 2023-03-09 09:50 .. -rw-r----- 1 1000 users 491 2023-05-08 10:00 vsphere-overlay-dns-control-plane.yaml -rw------- 1 root root 8392 2023-03-09 09:50 vsphere-overlay.yaml -rw-r----- 1 1000 users 484 2023-05-08 10:01 vsphere-overlay-dns-workers.yaml
3e: At POD - Update the file “config_default.yaml” for the below parameters with the correct .local domain under the directory “/root/.config/tanzu/tkg/providers/”:
# cd /root/.config/tanzu/tkg/providers/ root [ ~/.config/tanzu/tkg/providers ]# cat config_default.yaml | grep NODE_SEARCH_DOMAINS CONTROL_PLANE_NODE_SEARCH_DOMAINS: WORKER_NODE_SEARCH_DOMAINS: root [ ~/.config/tanzu/tkg/providers ]# sed -i "s/CONTROL_PLANE_NODE_SEARCH_DOMAINS:/CONTROL_PLANE_NODE_SEARCH_DOMAINS: vmware.local/g" config_default.yaml root [ ~/.config/tanzu/tkg/providers ]# sed -i "s/WORKER_NODE_SEARCH_DOMAINS:/WORKER_NODE_SEARCH_DOMAINS: vmware.local/g" config_default.yaml root [ ~/.config/tanzu/tkg/providers ]# cat config_default.yaml | grep NODE_SEARCH_DOMAINS CONTROL_PLANE_NODE_SEARCH_DOMAINS: vmware.local WORKER_NODE_SEARCH_DOMAINS: vmware.local
Step 4: Now proceed with the v2 TKG workload cluster deployment.