Setting the CA Certificate for the Web job in Concourse Version 7 and Higher Enables mTLS
Article ID: 297239
Updated On:
Products
Concourse for VMware Tanzu
Issue/Introduction
Issue Summary:
Starting in Concourse BOSH Release version 7 and up, the Concourse Web job will enable mTLS if a CA certificate is configured inside the TLS properties for the Web job in the Concourse manifest file. This may render the web interface inaccessible unless a client certificate issued by the configured CA is presented by the browser. In most situations, enabling mTLS was not the desired result.
Symptoms:
This issue presents itself as the web interface becoming inaccessible following an upgrade from an earlier version of Concourse to version 7 and above. The deployment will otherwise look healthy, with all instances showing as running when examined via the BOSH CLI.
This symptom may manifest as the Experiencing Turbulence error message appearing in the browser, but this is due to the interface being cached in the browser. If the cache is cleared, or the interface accessed via a private browsing session, the interface will simply fail to load. Due to this issue being cause by a failed TLS handshake, no HTTP error message will be observed. Examining the network console in the developer tools will show an error of SSL_ERROR_BAD_CERT_ALERT. The curl command may also be used to test for this issue and should respond with the following error:
Examining the web.stderr.log file from the web instance will show an error similar to the following:
Cause:
Following the release of Concourse 7, mTLS will be enabled if the CONCOURSE_TLS_CA_CERT setting is set in the configuration for the web job. In Concourse BOSH release 7.0 and above, this setting is enabled whenever the properties.tls.cert.ca is set in the properties for the web job in the BOSH manifest. This property can be quickly examined using the following BOSH CLI example:
If the ca: is present, then mTLS will be enabled.
The above output assumes that the manifest is stored fully interpolated, meaning that all variables have been replaced. In some deployments, this may not be the case and the output may appear similarly to the following:
In this case, the value for ((atc_tls)) will be substituted with the entire certificate from the variable store (either through a vars-store file or Credhub). This value will usually include the same ca: field and result in the same behavior.
Starting in Concourse BOSH Release version 7 and up, the Concourse Web job will enable mTLS if a CA certificate is configured inside the TLS properties for the Web job in the Concourse manifest file. This may render the web interface inaccessible unless a client certificate issued by the configured CA is presented by the browser. In most situations, enabling mTLS was not the desired result.
Symptoms:
This issue presents itself as the web interface becoming inaccessible following an upgrade from an earlier version of Concourse to version 7 and above. The deployment will otherwise look healthy, with all instances showing as running when examined via the BOSH CLI.
This symptom may manifest as the Experiencing Turbulence error message appearing in the browser, but this is due to the interface being cached in the browser. If the cache is cleared, or the interface accessed via a private browsing session, the interface will simply fail to load. Due to this issue being cause by a failed TLS handshake, no HTTP error message will be observed. Examining the network console in the developer tools will show an error of SSL_ERROR_BAD_CERT_ALERT. The curl command may also be used to test for this issue and should respond with the following error:
curl: (35) gnutls_handshake() failed: Certificate is bad
Examining the web.stderr.log file from the web instance will show an error similar to the following:
2021/05/21 21:21:21 http: TLS handshake error from 10.10.10.10:57962: tls: client didn't provide a certificate
Cause:
Following the release of Concourse 7, mTLS will be enabled if the CONCOURSE_TLS_CA_CERT setting is set in the configuration for the web job. In Concourse BOSH release 7.0 and above, this setting is enabled whenever the properties.tls.cert.ca is set in the properties for the web job in the BOSH manifest. This property can be quickly examined using the following BOSH CLI example:
# Replace <CONCOURSE MANIFEST> with the path to manifest file: bosh interpolate <CONCOURSE MANIFEST> --path=/instance_groups/name=web/jobs/name=web/properties/tls/cert ca: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- certificate: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- private_key: | -----BEGIN RSA PRIVATE KEY----- ... -----END RSA PRIVATE KEY----- Succeeded
If the ca: is present, then mTLS will be enabled.
The above output assumes that the manifest is stored fully interpolated, meaning that all variables have been replaced. In some deployments, this may not be the case and the output may appear similarly to the following:
bosh interpolate <CONCOURSE MANIFEST> --path=/instance_groups/name=web/jobs/name=web/properties/tls/cert ((atc_tls)) Succeeded
In this case, the value for ((atc_tls)) will be substituted with the entire certificate from the variable store (either through a vars-store file or Credhub). This value will usually include the same ca: field and result in the same behavior.
Environment
Product Version: 6.7
Resolution
Scenario 1: The Concourse manifest is not stored with variables
The bold section in the sample manifest below must be removed; then one has to re-deploy concourse.
The bold section in the sample manifest below must be removed; then one has to re-deploy concourse.
---
instance_groups:
- azs:
- AZ-1
- AZ-2
- AZ-3
instances: 1
jobs:
- name: bpm
release: bpm
- name: web
properties:
add_local_users:
- [REDACT]:[REDACT]
auth_duration: 72h
bpm:
enabled: true
build_log_retention:
default_builds: 200
maximum_builds: 250
build_tracker_interval: 60s
cluster_name: Example Cluster
...
tls:
bind_port: 443
cert:
ca: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
certificate: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
private_key: |
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
token_signing_key:
...
Scenario 2: The Concourse manifest has the properties.tls.cert field set to something similar to ((atc_tls))
In this scenario, this field of the manifest must be updated to explicitly declare the use of the certificate and private key in order to prevent the CA certificate from being added. Please update this field to look similar to the following:
tls:
bind_port: 443
cert:
certificate: ((atc_tls.certificate))
private_key: ((atc_tls.private_key))
Scenario 3: Concourse is being deployed using BOSH operations files to enable TLS
It is a common practice to deploy Concourse using the operations files from the Concourse BOSH Deployment, which are located in the cluster/operations directory of the downloadable package. Starting with v7.3.0 of the Concourse BOSH Deployment, the tls.yml file was updated such that mTLS would not be mistakenly enabled when trying to enable TLS.
When upgrading Concourse using these operations files, it is important to make sure that the operations files being used are for the appropriate version of Concourse.
Feedback
Yes
No
Powered by
