Pods Unable to Resolve External Corporate FQDNs in vSphere Kubernetes Service

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Application pods running inside a vSphere Kubernetes Service (VKS) cluster fail to resolve external or enterprise internal Fully Qualified Domain Names (FQDNs). Standard DNS queries inside pods return NXDOMAIN (Non-Existent Domain) or timeout errors.
  
  
• The underlying VKS worker node can successfully resolve the exact same FQDN via standard node-level diagnostics (e.g., dig or nslookup run directly on the host shell). Public external domains may or may not resolve depending on the upstream forwarder setup, but custom corporate subdomains consistently fail.
  
  
• When spawning a test pod inside a namespace enforced with Restricted Pod Security Standards, running nslookup reproduces the error:

~ $ nslookup <HOSTNAME>
Server:    <IP>
Address:   <IP>:53

** server can't find <HOSTNAME>: NXDOMAIN

Environment

• VMware vSphere Kubernetes Service

Cause

• A cluster domain collision exists combined with restricted fallthrough in the CoreDNS ConfigMap. When the Kubernetes cluster domain matches the corporate root domain, CoreDNS acts as the authoritative authority for that zone. Because the kubernetes plugin block limits fallthrough strictly to reverse lookup zones, queries for non-cluster enterprise endpoints are intercepted, find no internal service, and immediately terminate with NXDOMAIN.

Resolution

 

1. Inspect the current CoreDNS ConfigMap to confirm if fallthrough inside the kubernetes block is restricted or missing: kubectl get configmap coredns -n kube-system -o yaml

2. SSH into any VKS worker node and identify the active corporate upstream DNS server IPs (if any) (ignore loopback addresses like 127.0.0.1 or 127.0.0.53): cat /etc/resolv.conf

3. Edit the coredns ConfigMap in the kube-system namespace: kubectl edit configmap coredns -n kube-system
   
   Modify the Corefile section to enable unconditional fallthrough inside the kubernetes block and replace /etc/resolv.conf in the forward block with the explicit corporate DNS server IPs (if any):
   
   apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
            lameduck 5s
        }
        ready
        kubernetes <HOSTNAME> in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough  # <---- This allows everything to be forwarded to next endpoint if CoreDNS cannot respond to the DNS query
            ttl 30
        }
        hosts {
            <IP> <HOSTNAME>
            fallthrough
        }
        prometheus :9153
        forward . <IP> <IP> {
            max_concurrent 1000
        }
        cache 30 {
            disable success <HOSTNAME>
            disable denial <HOSTNAME>
        }
        loop
        reload
        loadbalance
    }

4. Restart the CoreDNS Deployment to force an immediate update: kubectl rollout restart deployment coredns -n kube-system

Additional Information

• To test DNS resolution safely inside a namespace enforced with Restricted Pod Security Standards, a security context override can be used:

kubectl run dns-test --image=busybox:1.36 -it --rm --restart=Never \
  --overrides='{
    "spec": {
      "securityContext": {
        "runAsNonRoot": true,
        "runAsUser": 1000,
        "runAsGroup": 1000,
        "fsGroup": 1000,
        "seccompProfile": { "type": "RuntimeDefault" }
      },
      "containers": [{
        "name": "dns-test",
        "image": "busybox:1.36",
        "command": ["sh"],
        "stdin": true,
        "tty": true,
        "securityContext": {
          "allowPrivilegeEscalation": false,
          "capabilities": { "drop": ["ALL"] }
        }
      }]
    }
  }'