DNS timeouts occurring during pipeline execution on Concourse Deployment

Products

• Concourse for VMware Tanzu

Issue/Introduction

This KB article goes over the scenario in which one experiences DNS timeouts while executing Concourse Pipelines. Symptoms of this issue might include sluggishness in pipeline execution speeds.

Cause

By default for Concourse, the containers within the workers are not allowed to access the host network directly, which will deny access to BOSH DNS. 

Resolution

There are a few workarounds available for this issue: 

Workaround 1: Define custom DNS servers via an Opsfile and redeploy the Concourse deployment. We can use either this Opsfile for the garden runtime or this Opsfile for the containerd runtime, create a var file for the chosen Opsfile via these steps below:

1. Generate the manifest of the current Concourse deployment

bosh -d <CONCOURSE DEP> manifest > concourse_manifest.yml

2. Create an ops file to store the desired DNS servers to use. As an example, we will use 8.8.8.8 and 8.8.4.4, however you can specify the DNS servers that you desire in place of these values. We can name this file dns-var.yml: 

garden_dns_servers:
  - 8.8.8.8
  - 8.8.4.4

3. We also create a file called garden-dns.yml which is taken from this Opsfile for the garden runtime. For this example, we will use garden-dns.yml which contains the following contents: 

- type: replace
  path: /instance_groups/name=worker/jobs/name=worker/properties/garden?/dns_servers?
  value: ((garden_dns_servers))

4. Redeploy concourse deployment with the Opsfile and var file: 

bosh deploy -d <CONCOURSE DEP> concourse_manifest.yml -o garden-dns.yml -l dns-vars.yml

5. Observe deployment to see if the timeouts still persist 

 

 

Workaround 2: If there is an explicit need for BOSH DNS, one can allow host access