intermittent issue resolving the FQDN

Products

• VMware Telco Cloud Automation

Issue/Introduction

Intermittent issue resolving the FQDN "FQDN.example.org" from pod "test-pod-coredns" towards DNS Server 

On looking at the pcap, it was seen that Total 15/5 times failed. One of the cluster is working fine while the other is having issues.

Environment

3.2

Cause

Trace path revealed that incase of working cluster, the T0 gateway routes to two different IPs  where as  the problematic cluster does not.
This is not the TCA or TKG issue. Issue needs to be checked with the design team from routing perspective.

Resolution

To diagnose the issue:

• Check DNS Lookup Inside the Pod
  kubectl exec -it  -- nslookup <fqdn>
  
  
• Test connectivity to DNS:
  kubectl exec -it <pod Name> -- ping <dns-ip>
  
  
• cat /etc/resolv.conf
  check that nameservers point to the cluster DNS IP
  
  
• Check logs and pcap from the dns server side to see why there is timeout
  
  
• Compare the tracepath from the working and non-working cluster