How to rotate expired atc_ca and atc_tls certificates of a Concourse deployment

Products

Concourse for VMware Tanzu

Issue/Introduction

PLEASE NOTE: The procedure of utilizing the Credhub CLI for rotating these certificates is only necessary for Concourse deployments running under an open source BOSH director. For Concourse deployments running under an Opsman managed BOSH Director, please see the following knowledge base article to rotate the atc_ca and atc_tls certificates using Credhub Maestro:

How to rotate Concourse atc_ca and postgress_ca using Maestro

IMPORTANT: It is necessary to understand how Concourse is deployed in your environment or foundation prior to following the certificate rotation procedure documented in this article.

Background

There are two CredHub processes that we will need to consider:

CredHub running on the BOSH Director VM.
CredHub running on the Web VM of your Concourse deployment.

This Knowledge Base (KB) article assumes that atc_ca and atc_tls certificates are kept in CredHub running on the BOSH Director and those certificates are now expired.

Symptoms

The following errors are seen if the Concourse pipeline with resource checks and builds is failing:

- /usr/local/bin/credhub login --client-name <redacted> --client-secret <redacted>
UAA error: Post https:<IP-or-URL-of-web-VM>:8844/oauth/token: x509: certificate has expired or is not yet valid

- /usr/local/bin/credhub get -n /bosh-master/concourse/atc_ca -k ca
You are not currently authenticated. Please log in to continue.

Environment

Product Version: 4.2
OS: Linux

Resolution

Pre-checks - SSH to Operations Manager VM

Confirm the following before moving forward:

Concourse is installed using a BOSH Director VM deployed by Operations Manager.
CredHub process on BOSH Director VM is running and you can target and login to that CredHub. Follow Scenario 1 from the following KB article: https://knowledge.broadcom.com/external/article?articleId=297967

In case the link above is broken, you can refer to the following commands:

Note: These commands will work from the Operations Manager VM.

Example Credhub target and login commands: 
(Credentials can be found here Bosh Commandline Credentials" from (Ops Manager UI -> Ops Manager/Director tile -> Credentials tab -> Bosh Commandline Credentials)

Commands:
- export BOSH_CLIENT=ops_manager BOSH_CLIENT_SECRET=<redacted> BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate BOSH_ENVIRONMENT=<redacted>

- export CREDHUB_CLIENT=$BOSH_CLIENT CREDHUB_SECRET=$BOSH_CLIENT_SECRET

- credhub api -s $BOSH_ENVIRONMENT:8844 --ca-cert $BOSH_CA_CERT

- credhub login

Verify that bosh deploy applies the tls-vars.yml ops file. For example:

Verify if both atc_ca and atc_tls certificates are expired by running the following CredHub get and OpenSSL commands:

get the ATC CA certificate and copy the certificate in ca block to a file atc_ca.crt by running the following command:

 - credhub get -n /p-bosh/<concourse-deployment-name>/atc_ca -k ca > atc_ca.crt

Check the certificate expiry and validity section to confirm that the certificate is expired:

- openssl x509 -in atc_ca.crt -noout -text

get the ATC TLS certificate and copy the certificate in certificate block to a file atc_tls.crt by running the following command:

- credhub get -n /p-bosh/<concourse-deployment-name>/atc_tls -k certificate > atc_tls.crt

Check the certificate expiry and validity section to confirm that the certificate is expired:

- openssl x509 -in atc_tls.crt -noout -text

Procedure

1. Once you have determined that both the atc_ca and atc_tls certificates are expired, we can move forward with regenerating those certificates in the following order:

atc_ca ---> atc_tls

2. The following CredHub command will regenerate the atc_ca certificate:

credhub regenerate -n /p-bosh/<concourse-deployment-name>/atc_ca

3. After the command above executes successfully, move on to regenerate the atc_tls certificate using the following command:

credhub regenerate -n /p-bosh/<concourse-deployment-name>/atc_tls

The next step is to run a run bosh deploy by extracting the manifest of your Concourse deployment from BOSH and add the --recreate option to it. Follow the steps below to do this:

Get the Concourse manifest:

bosh -d <concourse-deployment-name> manifest > concourse-deployment-manifest.yml

Re-deploy with --recreate flag:

bosh -d <concourse-deployment-name> deploy concourse-deployment-manifest.yml --recreate

4. Finally, once the bosh deploy succeeds, run a test pipeline in Concourse to do a sanity check and then run the pipeline(s) that were throwing errors before.

Note: Remember to update the locations if you have kept the same certs somewhere else to successfully unblock any stuck pipelines. For example, CredHub running in a Concourse Web VM instance.