Validating TLS connections in App Metrics v2.x

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Overview

This KB article goes over the steps to validate the TLS protocol versions that are being used during TLS connections on the db-and-errand-runner VM of the App Metrics tile. If the App Metrics deployment is being scanned by a vulnerability scanner (such as Rapid7 or Nessus), an alert may be raised regarding the fact that while TLS v1.2 is being used for Postgres database connections by default (which started in App Metrics v2.1.4 as mentioned here in the documentation), TLS v1.0 or TLS v1.1 can still be used during database connections, which are deprecated versions of the TLS protocol. 

Root Cause

The flagging via vulnerability scanner occurs because App Metrics version 2.2.0 and below use the Bosh Postgres release that contain versions of Postgres under version 12, which lack the functions to specify minimum allowed TLS protocol versions. Currently, the Bosh Postgres release being used by App Metrics v2.2.0 is currently v44 which we can see here. The Postgres version being used is also set at 11.15

The functions  ssl_min_protocol_version and ssl_max_protocol_version included in versions at or beyond Postgres version 12 allow for the ability to specify minimum supported TLS protocol versions as well as maximum supported TLS protocol versions. Because Postgres versions less than 12 lack these 2 functions, the App Metrics deployment (specifically the db-and-errand-runner VM)  may be flagged by a vulnerability scanner because TLS v1.0 and v1.1 can be used to connect in conjunction to v1.2 for connections. 

 

Validating TLS Connections

We can validate the TLS version being used in App Metrics with the following methods: 

Method 1
Checking the TLS_VERSION environment variable of the App Metrics application. We can do this by following the steps below: 

1. Target the App Metrics org and space

cf target -s app-metrics-v2

2. Print out the TLS_VERSION environment variable from the App Metrics application

cf env appmetrics

# Grep for TLS_VERSION
cf env appmetrics | grep TLS_VERSION

As we can see below, App Metrics is using TLS Version 1.2: 


Method 2
Log into the Postgres Database on the db-and-errand-runner VM, and view the TLS connections being used. We can do this by following the steps below: 

1. Navigate to the Appmetrics application org and space:

cf target -o system -s app-metrics-v2

2. Print out the App Metrics application environment variables. Take note of the POSTGRES_CLIENT_CERT and POSTGRES_CLIENT_KEY variables, and save these to a file named pgclient.crt and pgclient.key respectively: 

cf env appmetrics

3. SSH into the db-and-errand-runner VM:

bosh -d `bosh ds --column=name | grep ^appMetrics-` ssh db-and-errand-runner

4. Become root:

sudo -i

5. Save POSTGRES_CLIENT_CERT and POSTGRES_CLIENT_KEY cert client and cert key found in step 2 to files /tmp/pgclient.crt and /tmp/pgclient.key respectively:

db-and-errand-runner/1363f123-90a3-47f6-bade-fe93ba975d7b:~# vim /tmp/pgclient.crt
db-and-errand-runner/1363f123-90a3-47f6-bade-fe93ba975d7b:~# vim /tmp/pgclient.key
db-and-errand-runner/1363f123-90a3-47f6-bade-fe93ba975d7b:~#

6. Apply restrictive permissions to the client key:

chmod 600 /tmp/pgclient.key

7. Log into the Postgres Database using the pgclient.crt and pgclient.key:

/var/vcap/packages/postgres-11.9/bin/psql "port=5432 host=localhost user=pgadmin sslcert=/tmp/pgclient.crt sslkey=/tmp/pgclient.key sslmode=require dbname=indicators"

NOTE: The Postgres package version may be different if you have an AppMetrics version that isn't 2.1.5. In our case, it happens to be postgres-11.9 You may need to validate which postgres package is being used by listing the postgres package via the command ls -ltr /var/vcap/packages/postgres-* to find the Postgres package being used. Usually we would want to use the highest version. So if we see something similar to the output in the screenshot below, we would would opt to use postgres-11.9 instead of postgres-9.6.10



8. Query for connection information:

SELECT datname,usename, ssl, version, client_addr
FROM pg_stat_ssl
JOIN pg_stat_activity
ON pg_stat_ssl.pid = pg_stat_activity.pid;

Example output:

indicators=# SELECT datname,usename, ssl,version, client_addr
FROM pg_stat_ssl
JOIN pg_stat_activity
ON pg_stat_ssl.pid = pg_stat_activity.pid;
datname | usename | ssl | version | client_addr
------------+---------+-----+---------+---------------
| | f | |
| vcap | f | |
indicators | pgadmin | t | TLSv1.2 | 10.225.29.169
indicators | pgadmin | t | TLSv1.2 | 10.225.29.169
indicators | pgadmin | t | TLSv1.2 | 10.225.29.169
indicators | pgadmin | t | TLSv1.2 | 127.0.0.1
| | f | |
| | f | |
| | f | |
(9 rows)
 
indicators=#

Environment

Product Version: 2.11

Resolution

Currently, there is no permanent fix for the App Metrics db-and-errand-runner VM being able to use TLS v1.0 and v1.1 during connections. 

The permanent fix would be for a future release of App Metrics to include the Postgres Bosh release v45 which contains Postgres 15.3, which is a version of Postgres that includes the functions  ssl_min_protocol_version and ssl_max_protocol_version which should prevent the use of TLS v1.0 and TLS v1.1 during connections. 

This issue has been raised with the product team, and anticipate a fix in the next version of the App Metrics tile.