Tanzu Hub 10.3.x Installation Fails on wait-ops-kafka Init Container Due to Reverse-DNS Lookups

Products

• VMware Tanzu Platform - Hub

Issue/Introduction

During installation or reconciliation of VMware Tanzu Platform - Hub 10.3.x, the wait-ops-kafka init container on an ops-kafka broker pod fails or retries repeatedly. Pod logs show Jolokia health-check requests (e.g. curl http://ops-kafka-controller-headless:8778/jolokia/read/kafka.controller:type=KafkaController,name=ActiveControllerCount) taking significantly longer than expected, exceeding the init container's timeout.

Environment

- VMware Tanzu Platform - Hub
- Tanzu Platform

Cause

Kafka's Jolokia HTTP agent performs a reverse DNS lookup of the calling pod's IP address as part of its host-based access check. If the calling pod is not yet registered in DNS, the lookup falls through CoreDNS to the upstream DNS resolver. In environments where that resolver takes several seconds to respond, this exceeds the wait container's timeout.

Resolution

Apply two changes: a CoreDNS patch that rejects unresolvable reverse lookups for the cluster's own Pod/Service CIDR immediately, and a ytt overlay that lowers ndots for Kafka's workloads.

Step 1: Identify the cluster's Pod CIDR and Service CIDR:

kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{end}'

kubectl -n default get svc kubernetes -o jsonpath='{.spec.clusterIP}{"\n"}'

Step 2: Edit the coredns ConfigMap in kube-system, adding a template block for the derived reverse zones inside the existing .:53 server block, after the kubernetes plugin and before forward (example zones shown for Pod CIDR 10.200.0.0/16 and Service CIDR 10.100.200.0/24 - substitute your own):

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
          ttl 30
        }
        prometheus :9153
        template ANY PTR 200.10.in-addr.arpa. 200.100.10.in-addr.arpa. {
            rcode NXDOMAIN
        }
        forward . /etc/resolv.conf {
          policy sequential
        }
        cache 30
        loop
        reload
        loadbalance
    }

Step 3: Create a ytt-overlay Secret lowering ndots to 2 for Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs, and annotate the top-level sm PackageInstall to apply it:

kubectl apply -n "$NAMESPACE" -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: ndots-pod-overlay-secret
stringData:
  ndots-overlay.yaml: |
    #@ load("@ytt:overlay", "overlay")
    #@ kinds = ["Deployment", "StatefulSet", "DaemonSet", "Job"]
    #@ for kind in kinds:
    #@overlay/match by=overlay.subset({"kind": kind}), expects="0+"
    ---
    spec:
      #@overlay/match missing_ok=True
      template:
        #@overlay/match missing_ok=True
        spec:
          #@overlay/match missing_ok=True
          dnsConfig:
            #@overlay/match missing_ok=True
            options:
              #@overlay/match by="name", missing_ok=True
              - name: ndots
                value: "2"
    #@ end
    #@overlay/match by=overlay.subset({"kind": "CronJob"}), expects="0+"
    ---
    spec:
      #@overlay/match missing_ok=True
      jobTemplate:
        #@overlay/match missing_ok=True
        spec:
          #@overlay/match missing_ok=True
          template:
            #@overlay/match missing_ok=True
            spec:
              #@overlay/match missing_ok=True
              dnsConfig:
                #@overlay/match missing_ok=True
                options:
                  #@overlay/match by="name", missing_ok=True
                  - name: ndots
                    value: "2"
EOF

 

kubectl apply -n "$NAMESPACE" -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: ndots-pkgi-overlay-secret
stringData:
  pkgi-overlay.yaml: |
    #@ load("@ytt:overlay", "overlay")
    #@overlay/match by=overlay.subset({"kind": "PackageInstall"}), expects="0+"
    ---
    metadata:
      #@overlay/match missing_ok=True
      annotations:
        #@overlay/match missing_ok=True
        ext.packaging.carvel.dev/ytt-paths-from-secret-name.99: ndots-pod-overlay-secret
EOF

 

kubectl annotate pkgi sm -n "$NAMESPACE" \
  ext.packaging.carvel.dev/ytt-paths-from-secret-name.99=ndots-pkgi-overlay-secret \
  --overwrite

 

Step 4: Confirm the fix:

kubectl exec -n <namespace> <kafka-pod> -c kafka-broker -- cat /etc/resolv.conf
# expect: options ndots:2

 

Both changes are not persisted across Hub Apply Change operations and must be reapplied after each reconciliation.

 

Additional Information

• Related: Tanzu Hub 10.3.x Installation Fails on temporal-schema Job