Spring Cloud Services Smoke Test Fails with Validator Exception error "PKIX path building failed"

Products

• VMware Spring Runtime (Core)

Issue/Introduction

Symptoms:

After enabling "Secure service instance credentials option" under Spring Cloud Services (SCS) config, Spring Cloud Services Smoke Test fails with the following error:

Error creating service instance: org.springframework.web.client.ResourceAccessException: 
I/O error on POST request for "https://credhub.service.cf.internal:8844/api/v1/data": 
sun.security.validator.ValidatorException: PKIX path building failed

Environment

Cause

This error occurs during SCS service instance creation, and it's from SCS Broker.

The SCS Broker requires a secure channel, so it talks to CredHub over HTTPS. The error "PKIX path building failed" indicates that HTTP client used on the SCS Broker does not trust the certificate presented by the CredHub server.

One possible reason for this is due to the old version of the JBP (Java buildpack) being used. We need at least version 4.1 of the JBP as this will include the "Container Security Provider" feature. The feature will automatically load certificates that are trusted by the platform into the JVM so that the JVM trusts them as well. This process resolves the issue because the platform provided certificates include the certificate authority that was used to sign the certificate presented by the CredHub server. Thus the SCS Broker will trust CredHub.

When you set the "TRUST_CERTS" environment variable for SCS Broker apps that would only partially work around this issue. Doing that would result in the "invalid_token" error below.

org.springframework.credhub.core.CredHubException: Error calling CredHub: 401: 
{"error":"invalid_token","error_description":
"Full authentication is required to access this resource"}

This happens because CredHub requires mutual TLS to be used so that it can identify a client application. To accomplish this, the SCS Broker needs to send its container identity certificate to CredHub. The container identity certificate contains authentication info for CredHub (specifically the GUID of the application running in the container). If CredHub does not get that in the request from SCS Broker, CredHub will fall back to looking for OAuth2 credentials in the header, and it would fail with HTTP 401 error code since the SCS Broker is not configured to do this. The resolution is also to use v4.1 or higher of the JBP which includes the "Container Security Provider" feature. With this feature, the JVM will automatically send the SCS Broker's container identity certificate, and this satisfies the authentication that CredHub requires.

Resolution

The only way to get all this working is to get the SCS Broker pushed with a version of JBP that has the "Container Security Provider" feature in it, and the feature is introduced in JBP v4.1.  

It is recommended to upgrade to the latest version of Spring Cloud Services to get the latest Java Buildpack. If this is not possible, a Java Buildpack version of 4.1 or higher can be manually added to the foundation and configured for use by the SCS broker. Instructions for setting this can be found here.