Credhub Interpolate Tasks Fail with x509 Authority Error
search cancel

Credhub Interpolate Tasks Fail with x509 Authority Error

book

Article ID: 378285

calendar_today

Updated On:

Products

Concourse for VMware Tanzu

Issue/Introduction

Following a rotation of the Concourse ATC CA certificate, credhub-interpolate tasks begin to fail with an error similar to the following:

Get "https://CONCOURSEDOMAIN:8844/info": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "atcCA")

Cause

The above error occurs in Concourse deployments with a colocated Credhub/UAA when the CA certificate used in the credhub-interpolate task is not properly trusted. This typically occurs following a rotation of the atc_ca certificate, either manual or as part of a full CA rotation procedure of the Ops Manager hosting the deployment. In the aforementioned colocated Credhub/UAA configuration, the Credhub and UAA deployments both utilize the atc_tls certificate used by the web instance which is signed by the atc_ca certificate. The atc_tls certificate is rotated as part of the atc_ca certificate rotation procedure. If the atc_ca certificate is not then exported from the BOSH director Credhub and appropriately configured in the task, or added to the BOSH Director trusted certificates, then the interpolate tasks may fail.

Resolution

To resolve this, you will first have to determine how the certificate is being trusted by the credhub-interpolate task. The following three scenarios outline how this CA certificate can be passed to the task:

  1. As plain text via the task configuration
    1. When examining the YAML pipeline configuration for the impacted pipeline, you will see the CREDHUB_CA_CERT parameter set to multiline text containing the PEM encoded certificate
  2. Using an interpolated value in the task configuration
    1. When examining the YAML pipeline configuration for the impacted pipeline, you will see the CREDHUB_CA_CERT parameter set a variable (e.g ((credhub_ca)) )
  3. Via the BOSH Director trusted certificates
    1. When examining the YAML pipeline configuration for the impacted pipeline,  the CREDHUB_CA_CERT parameter will not be set

Please Note: There is one additional scenario that will not be explicitly covered in this article. That scenario involves the credhub-interpolate task pointing to a Credhub instance external to the Concourse deployment. In this scenario, the below instructions should still prove useful for determining how to resolve the issue. Additionally, as this article assumes a colocated Credhub it also assumes that the credential manager configured for Concourse is also Credhub. If the deployment is utilizing Vault, please refer to the documentation for that product to determine the appropriate instructions for replacing the certificate.

For every scenario, we will first need to extract the new certificate from the BOSH Director Credhub. Below are the instructions for this process:

  1. Log into the Ops Manager UI
  2. Navigate to the BOSH Director tile > Credentials > Bosh Commandline Credentials
  3. Leave this window open
  4. SSH into the Ops Manager VM
  5. Export the credentials located in step 2, as an example:
    export BOSH_CLIENT=ops_manager BOSH_CLIENT_SECRET=SECRET BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate BOSH_ENVIRONMENT=255.255.255.255 bosh
  6. Next we will use the above exported credentials to export the Credhub CLI credentials:
    export CREDHUB_SERVER="$BOSH_ENVIRONMENT:8844" CREDHUB_CLIENT="$BOSH_CLIENT"  CREDHUB_SECRET="$BOSH_CLIENT_SECRET" CREDHUB_CA_CERT="$BOSH_CA_CERT"
  7. Use the Credhub CLI to extract the atc_ca certificate to a file named atc_ca_certificate.pem (Please note that CONCOURSEDEPLOYMENT should be replaced with the name of Concourse BOSH deployment)
    credhub get -n /p-bosh/CONCOURSEDEPLOYMENT/atc_ca -k certificate > atc_ca_certificate.pem
  8. Preserve the resulting atc_ca_certificate.pem file for later use

 

Now that we have the new atc_ca certificate exported, we can move on to resolving the issue. Below are the instructions for each scenario outlined earlier: 

 

Scenario 1 - Plain text via the task configuration:

  1. Open the pipeline configuration file for editing. Many organizations use version control to manage their pipeline configurations, so please follow any internal change control processes for editing the configuration
  2. Locate the credhub-interpolate task
  3. Locate the CREDHUB_CA_CERT parameter for the task
  4. Replace the block text with the contents of the atc_ca_certificate.pem file, making sure to properly format the spacing
  5. Repeat the first four steps for all other impacted tasks in this pipeline
  6. Re-fly the pipeline
  7. Re-run the impacted pipeline to verify that the issue has been resolved

    Please Note: These steps will need to be repeated for all impacted pipelines.

 

Scenario 2 - Using an interpolated value:

This scenario will require the use of the Credhub CLI located on a system with access to the Credhub server configured in the credhub-interpolate task as well at the atc_ca_certificate.pem file that was previously exported.

  1. Examine the credhub-interpolate task in the pipeline configuration file to determine the name of the value being interpolated. For the purposes of this article, we will assume a name of credhub_ca
  2. Using the BOSH CLI, retrieve the Credhub client information from a Concourse web VM:
    bosh -d CONCOURSEDEPLOYMENT ssh web/0 'sudo grep CREDHUB /var/vcap/jobs/web/config/bpm.yml'

# Sample output

% bosh -d CONCOURSEDEPLOYMENT ssh web/0 'sudo grep CREDHUB /var/vcap/jobs/web/config/bpm.yml'
Using environment '255.255.255.255' as client 'ops_manager'

Using deployment 'CONCOURSEDEPLOYMENT'

Task 4710. Done
web/12345678-1234-1234-1234-123456789012: stderr | Unauthorized use is strictly prohibited. All access and activity
web/12345678-1234-1234-1234-123456789012: stderr | is subject to logging and monitoring.
web/12345678-1234-1234-1234-123456789012: stdout |     CONCOURSE_CREDHUB_CLIENT_ID: "concourse_to_credhub_client"
web/12345678-1234-1234-1234-123456789012: stdout |     CONCOURSE_CREDHUB_CLIENT_SECRET: "SECRET"
web/12345678-1234-1234-1234-123456789012: stdout |     CONCOURSE_CREDHUB_PATH_PREFIX: "/concourse"
web/12345678-1234-1234-1234-123456789012: stdout |     CONCOURSE_CREDHUB_CA_CERT: "/var/vcap/jobs/web/config/env/CONCOURSE_CREDHUB_CA_CERT"
web/12345678-1234-1234-1234-123456789012: stdout |     CONCOURSE_CREDHUB_INSECURE_SKIP_VERIFY: "false"
web/12345678-1234-1234-1234-123456789012: stdout |     CONCOURSE_CREDHUB_URL: "https://255.255.255.254:8844"
web/12345678-1234-1234-1234-123456789012: stderr | Connection to 255.255.255.254 closed.

Succeeded
  3. Record the values for CONCOURSE_CREDHUB_CLIENT_ID, CONCOURSE_CREDHUB_CLIENT_SECRET, and CONCOURSE_CREDHUB_URL.
  4. Use these values to export the Credhub CLI credentials for the Concourse Credhub instance into the shell environment:
    export CREDHUB_CLIENT=CONCOURSE_CREDHUB_CLIENT_ID CREDHUB_SECRET=CONCOURSE_CREDHUB_CLIENT_SECRET CREDHUB_SERVER=CONCOURSE_CREDHUB_URL CREDHUB_CA_CERT=/path/to/atc_ca_certificate.pem
  5. Next, use the Credhub CLI to find the path for the credential using the variable found in the task configuration in step 1:
    credhub find -n credhub_ca

     

    Please Note: This command may return several results. Concourse will search for variables in the configured prefix (By default: /concourse), followed by the team name and the pipeline name. If it is unable to locate the variable at this path, it will then look in the prefix and team name. This allows for variables to be scoped either at the pipeline or team level. For example, you may see both /concourse/main/credhub_ca for a variable scoped for the main team, and /concourse/main/foundation_management/credhub_ca for one scoped for the foundation_management pipeline in the main team. For the purposes of this article we will assume that it is team scope to avoid unnecessary ambiguity, but please keep this information in mind when following these instructions.

  6. Once we have the path for our variable, we can set the value to the new atc_ca certificate:
    credhub set -n /concourse/main/credhub_ca -t certificate -c /path/to/atc_ca_certificate.pem
  7. Re-run the impacted pipeline(s) to verify that the issue has been resolved

    Please Note: If the variable was pipeline scoped and other pipelines are performing interpolate tasks using the same Credhub server, then step 6 may need to be repeated for each variable in order to resolve this issue for all pipelines.

 

Scenario 3 - Via the BOSH Director trusted certificates:

The worker nodes in the Concourse deployment are able to consume certificates that are configured to be trusted by the BOSH Director. In this scenario, there will be no CREDHUB_CA_CERT parameter set because the certificate is implicitly trusted by the worker. When the atc_ca certificate is rotated, the new version will need to be added to the BOSH Trusted Certificates.

  1. Log into the Ops Manager UI
  2. Navigate to BOSH Director tile > Security
  3. Paste the contents of the atc_ca_certificate.pem file into the Trusted Certificates section below all existing certificates
  4. Click Save
  5. Perform an Apply Changes on the BOSH Director tile
  6. Using the BOSH CLI, export the existing Concourse deployment manifest:
    bosh -d CONCOURSEDEPLOYMENT manifest > concourse_manifest.yml
  7. Perform a deploy using the exported manifest:
    bosh -d CONCOURSEDEPLOYMENT deploy concourse_manifest.yml
  8. Once the deploy is done, re-run the impacted pipeline(s) to verify that the issue has been resolved

Additional Information

Please see the following for additional information:

Powered by Wolken Software