Tanzu Hub 10.3.x Installation Fails on temporal-schema Job

Products

• VMware Tanzu Platform - Hub

Issue/Introduction

When installing or upgrading VMware Tanzu Platform - Hub 10.3.x, the installation fails during the reconciliation of the temporal package. Specifically, the temporal-schema-1 job remains in a Pending or CrashLoopBackOff state.

Reviewing the logs for the failing pod reveals a timeout error in the create-default-namespace init container:

time=2026-02-05T10:07:22.747 level=ERROR msg="failed reaching server: context deadline exceeded"

Environment

Tanzu Hub 
Tanzu Platform 

Cause

The issue is caused by a DNS resolution failure within the Kubernetes cluster. The temporal-schema job attempts to connect to the temporal server using a short Fully Qualified Domain Name (FQDN), such as temporal-frontend:7233.

Due to the default Kubernetes dnsConfig setting of ndots:5, the resolver performs multiple unnecessary recursive lookups for short names, leading to a "context deadline exceeded" error. Connectivity is successful only when using the full FQDN (e.g., temporal-frontend.tanzusm.svc.cluster.local).

Resolution

To resolve this issue, apply a ytt overlay to the temporal package to lower the ndots value and force the use of the full FQDN for the temporal address.

Step 1: Create the Overlay Secret

Create a file named temporal-overlay.yaml with the following content to patch the DNS configuration and environment variables of the temporal-schema-1 job.

apiVersion: v1
kind: Secret
metadata:
  name: temporal-schema-overlay
  namespace: tanzusm
stringData:
  patch.yaml: |
    #@ load("@ytt:overlay", "overlay")

    #@overlay/match by=overlay.subset({"kind": "Job", "metadata": {"name": "temporal-schema-1"}})
    ---
    metadata:
      #@overlay/match missing_ok=True
      annotations:
        #@overlay/match missing_ok=True
        kapp.k14s.io/update-strategy: "always-replace"
    spec:
      template:
        spec:
          #@overlay/match missing_ok=True
          dnsConfig:
            options:
              - name: ndots
                value: "1"
          initContainers:
            #@overlay/match by="name"
            - name: create-default-namespace
              env:
                #@overlay/match by="name"
                - name: TEMPORAL_ADDRESS
                  value: "temporal-frontend.tanzusm.svc.cluster.local.:7233"
            #@overlay/match by="name"
            - name: create-custom-search-attributes
              env:
                #@overlay/match by="name"
                - name: TEMPORAL_ADDRESS
                  value: "temporal-frontend.tanzusm.svc.cluster.local.:7233"

Step 2: Apply Overlay to the PackageInstall

Pause the package reconciliation and annotate the PackageInstall to include the new overlay.

# Pause the package reconciliation
kctrl package installed pause -i temporal -n tanzusm --yes

# Annotate the PackageInstall to use the overlay secret
kubectl -n tanzusm annotate packageinstalls temporal ext.packaging.carvel.dev/ytt-paths-from-secret-name.7=temporal-schema-overlay

# Unpause/Kick the package reconciliation to apply changes
kctrl package installed kick -i temporal -n tanzusm --yes

Step 3: Verification

Monitor the temporal-schema-1 job. The overlay will cause kapp to replace the existing job with the new specification. The job should now successfully reach the temporal server and complete the installation.

kubectl -n tanzusm get pods | grep temporal-schema

 

Additional Information

https://discuss.kubernetes.io/t/coredns-unnecessary-recursion-on-shortnames/23588