Global Trust Configuration: Propagating Internal CA Certificates to Concourse Containers via BOSH

Products

• Concourse for VMware Tanzu

Issue/Introduction

When running workloads inside Concourse CI/CD pipelines that communicate with internal, corporate, or self-signed secure endpoints (such as internal Git servers, private container registries, or custom artifact repositories), pipeline jobs or resource checks frequently fail during network handshakes. These failures occur even when network routes are wide open and user credentials are confirmed valid.

The typical symptom manifests in the Concourse Web UI or CLI logs as an explicit x509 validation failure, preventing the orchestration engine from executing actions on remote endpoints. For instance, a resource container trying to check a Git repository or pull an image will fail with error patterns such as:

fatal: unable to access 'https://internal-git.local/...': server certificate verification failed
or
error: failed to ping registry: x509: certificate signed by unknown authority

Attempting to fix this problem individually across pipelines by modifying local code or implementing individual bypass flags (such as skip_ssl_verification: true) creates massive administrative overhead, introduces severe security vulnerabilities, and is highly unscalable across an enterprise deployment managing hundreds of pipelines.

Cause

Concourse containers fail to trust internal endpoints because they inherit an outdated or un-synchronized certificate store bind-mounted directly from the underlying BOSH Worker VM host.

A standard bosh deploy doesn't automatically re-initialize this shared file volume, requiring an explicit --recreate deployment flag to clear stale runtime configurations and successfully cascade the new CA certificate chain into the container layer.

Resolution

To permanently and securely resolve this issue across all pipelines globally, the certificate authority trust chain must be structurally applied at the infrastructure layer, followed by a hard lifecycle recreation of the Concourse deployment.

Follow this strict technical sequence to establish global trust propagation:

Step 1: Inject Certificate into the BOSH Director Tile

1. Log in to your Ops Manager UI.

2. Navigate to the BOSH Director Tile and select the Security tab.

3. Locate the Trusted Certificates text block and securely append your internal Root/Intermediate CA certificates (in standard X.509 PEM format).

4. Click Save.

Step 2: Propagate Trust via BOSH Director Apply Changes

1. Return to the Ops Manager Installation Dashboard.

2. Click Review Pending Changes, select only the BOSH Director, and click Apply Changes.

3. This updates the OS-level stemcell trust store across the ecosystem, dropping the new keys into the host operating system path at /etc/ssl/certs.

Step 3: Regenerate the Concourse Deployment Manifest

1. From your infrastructure jumpbox, interface with the BOSH CLI and generate a fresh deployment manifest from your active ops files or template stubs.

2. Crucial: This action forces the manifest interpreter to pull the Director's newly active trusted_certs payload into the structural definition of the target Concourse deployment.
   
   

  bosh -e your-env -d concourse manifest > concourse_manifest.yml

Step 4: Execute a Hard Reconstruction Deploy

1. Deploy the newly generated manifest file back to the environment.

2. You must append the explicit --recreate flag to force BOSH to destroy the active worker instances and spin up brand-new VMs. Run the following command:

bosh -e your-env -d concourse deploy concourse_manifest.yml --recreate