"dial tcp: lookup TMC_DNS_ZONE on DNS_IP:53: no such host" and "secret <SECRET_NAME> not found" errors causing pod failures when installing an instance of VMware Cloud Director Extension for VMware Tanzu Mission Control

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• Installing VMware Cloud Director Extension for VMware Tanzu Mission Control through Cloud Director Solution Add-On Management.
• Pods in the tmc-local namespace on the target Kubernetes cluster have a CreateContainerConfigError status, for example:

NAMESPACE  NAME                             READY  STATUS                    
tmc-local  audit-service-consumer-<POD_ID>  0/1    CreateContainerConfigError
tmc-local  audit-service-consumer-<POD_ID>  0/1    CreateContainerConfigError
tmc-local  audit-service-server-<POD_ID>    0/1    CreateContainerConfigError
tmc-local  audit-service-server-<POD_ID>    0/1    CreateContainerConfigError
tmc-local  dataprotection-server-<POD_ID>   0/1    CreateContainerConfigError
tmc-local  dataprotection-server-<POD_ID>   0/1    CreateContainerConfigError
tmc-local  inspection-server-<POD_ID>       0/2    CreateContainerConfigError
tmc-local  inspection-server-<POD_ID>       0/2    CreateContainerConfigError

• Pods in the tmc-local namespace on the target Kubernetes cluster have a CreateContainerConfigError status and show events regarding missing secrets, for example the audit-service-consumer-<POD_ID> pods show events similar to the following:

Type     Reason       Age    From     Message
----     ------       ----   ----     -------
Warning  FailedMount  <AGE>  kubelet  MountVolume.SetUp failed for volume "kafka-root-ca" : secret "audit-server-kafka-creds" not found
Warning  FailedMount  <AGE>  kubelet  MountVolume.SetUp failed for volume "consumer-tls" : secret "audit-consumer-tls" not found
Warning  Failed       <AGE>  kubelet  Error: secret "audit-s3-creds" not found

• The container logs on the s3-access-operator-<POD_ID> pod in the tmc-local namespace has errors of the form:

<TIME_STAMP>    [30mERROR[0m    Reconciler error    {"controller": "s3accesspolicy", "controllerGroup": "infra.tmc.eng.vmware.com", "controllerKind": "S3AccessPolicy", "S3AccessPolicy": {"name":"inspection-inspection-s3","namespace":"tmc-local"}, "namespace": "tmc-local", "name": "inspection-inspection-s3", "reconcileID": "<RECONCILE_ID>", "error": "error in retrieving credentials value: RequestError: send request failed\ncaused by: Post \"https://<TMC_DNS_ZONE>/\": dial tcp: lookup <TMC_DNS_ZONE> on <DNS_IP>:53: no such host"}

Environment

VMware Cloud Director 10.x

Cause

This issue can occur if the DNS records for the DNS zone and the sub domains for Tanzu Mission Control have not been configured correctly before installing VMware Cloud Director Extension for VMware Tanzu Mission Control.

Resolution

Ensure that the DNS records for the DNS zone and all the sub domains for Tanzu Mission Control have been configured and DNS resolution is working for each.
Ensure DNS resolution is working on the Routed Organization VDC Network to which the target kubernetes cluster is connected.
Details of the list of DNS records required are in the VMware Cloud Director Extension for VMware Tanzu Mission Control documentation, Before you begin.