Java App Running on Cloud Foundry Reinitializes ContainerTrustManagerFactory on Every Connection

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

For Java applications running on Cloud Foundry, the Java buildpack will by default install the container security provider. This adds an additional TrustManagerFactory which enables Java applications to automatically trust certificates deployed to the foundation, including Bosh Trusted Certificates.

For well behaved applications and libraries, this happens seamlessly and does not impact applications. In some cases, applications and libraries have been observed to perform custom TLS initialization which can cause minor problems due to the way it interacts with the TrustManagerFactory installed by the container security provider.

VMware Support has observed this happen under a couple scenarios:

• Java application using HikariCP & the MySQL JDBC driver
• Java application using HikariCP & the Oracle JDBC driver

When the problem happens, you will notice the following log message occur every time a new connection to the database is created as opposed to just once when the application is first started.

2020-03-19T15:26:16.14-0400 [APP/PROC/WEB/0] OUT 2020-03-19 19:26:16.144  INFO 12 --- [onnection adder] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2020-03-19T15:26:16.34-0400 [APP/PROC/WEB/0] OUT 2020-03-19 19:26:16.343  INFO 12 --- [onnection adder] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager

The reason you see this message multiple times is because the JDBC drivers listed above are known to call TrustManagerFactory.init, which in turn calls .init on the container security provider's TrustManagerFactory, on every connection creation, as opposed to calling it once. This causes the container security provider's TrustManagerFactory to reinitialize itself, which includes logging the lines listed above.

Resolution

There are a couple possible resolutions here.

1. You can turn logging off. The container security provider's package is org.cloudfoundry.security, just set the level to WARN or ERROR for this package and that will hide the log lines emitted during initialization of the TrustManagerFactory.

   This can be done in the application, using your logging framework of choice. For Spring Boot applications, it is accomplished by setting the property logging.level.org.cloudfoundry.security=WARN.

   Note: This will not remove any performance overhead or prevent the creation of superfluous garbage due to reinitializing all TrustManagerFactories on every connection creation. This workaround only changes the log level which prevents the messages from being written. For what it's worth, this is not a problem specific to Cloud Foundry though. This reinitialization is happening in the JDBC driver, so that is happening whenever the JDBC drivers listed above are being used, not just when running on Cloud Foundry. The difference is that you can see what is happening on Cloud Foundry, due to the logging output of the container security provider's TrustManagerFactory.

2. Disable the container security provider. This will instruct the Java buildpack to not install the container security provider. This means that when the JDBC driver initializes the TrustManagerFactory chain, it won't include the container security provider's TrustManagerFactory, which in turn means you won't see the logging and all the work done by the container security provider's TrustManagerFactory will not happen. That said, the JDBC drivers listed above are still going to reinitialize every other TrustManagerFactory in the chain so there will still be extraneous work and garbage as a part of the reinitialization. In short, as long as you are using a JDBC driver that behaves in this manner there will be some inefficiency.

   Note: Disabling the container security provider means that your application will not get the benefits of it, which include automatically trusting Bosh deployed certificates, easy mTLS using Diego instance id certificates.

A final note: There is no option for a code fix in the container security provider. At the moment, the container security provider performs come caching of the certificates it loads. However, this can’t be cached across invocations of TrustManagerFactory.init. This is because the description of that method in the Javadocs for the interface instructs an implementor to Initializes this factory with a source of provider-specific key material. Caching across this method call would not make sense and defeat the purpose of calling the method.