"SunCertPathBuilderException: unable to find valid certification path to requested target" exception from Java when running application as a task in Tanzu Application Service for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

You have an application that runs successfully when you run it as a web application. The app can load the certificates correctly. Here is a sample log produced when you run your application run as an app. 

2021-10-07T14:04:20.219+08:00 [APP/PROC/WEB/0] [OUT] 2021-10-07 14:04:20.219 INFO 23 --- [ main] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager 2021-10-07T14:04:20.221+08:00 [APP/PROC/WEB/0] [OUT] 2021-10-07 14:04:20.221 INFO 23 --- [ main] oundryContainerKeyManagerFactory$SunX509 : Adding System Key Manager 2021-10-07T14:04:20.221+08:00 [APP/PROC/WEB/0] [OUT] 2021-10-07 14:04:20.221 INFO 23 --- [ main] oundryContainerKeyManagerFactory$SunX509 : Adding Key Manager for /etc/cf-instance-credentials/instance.key and /etc/cf-instance-credentials/instance.crt 2021-10-07T14:04:20.224+08:00 [APP/PROC/WEB/0] [OUT] 2021-10-07 14:04:20.223 INFO 23 --- [-instance.crt-0] org.cloudfoundry.security.FileWatcher : Start watching /etc/cf-instance-credentials/instance.crt 2021-10-07T14:04:20.225+08:00 [APP/PROC/WEB/0] [OUT] 2021-10-07 14:04:20.225 INFO 23 --- [-instance.key-0] org.cloudfoundry.security.FileWatcher : Start watching /etc/cf-instance-credentials/instance.key 2021-10-07T14:04:20.420+08:00 [APP/PROC/WEB/0] [OUT] 2021-10-07 14:04:20.419 INFO 23 --- [ main] o.c.s.FileWatchingX509ExtendedKeyManager : Initialized KeyManager for /etc/cf-instance-credentials/instance.key and /etc/cf-instance-credentials/instance.crt 

However, when you run the app as a task, you get the following output:

cf run-task runner ".java-buildpack/open_jdk_jre/bin/java org.springframework.boot.loader.JarLauncher" --name my-task

----------------------------------------------------------------------------------------

OUT 2021-10-07 14:42:52.549 INFO 15 --- [ main] c.p.p.p.c.g.u.w.RestServiceImpl : Invoking [POST] request to [https://another-microservice-endpoint] with request object: [{"headers":{"Content-Type":["application/json"],"site":["core"],"Authorization": ...."}"}] 752fae70-3650-4a23-9150-b806a8f54cb6 APP/TASK/3a966d54/0 2021-10-07T06:42:52.685675382Z OUT 2021-10-07 14:42:52.684 ERROR 15 --- [ main] c.p.p.p.c.g.u.w.RestServiceImpl : Encountered exception invoking [POST] request to [https://another-microservice-endpoint] APP/TASK/3a966d54/0 2021-10-07T06:42:52.688318757Z OUT 2021-10-07 14:42:52.687 INFO 15 --- [ main] c.p.p.p.c.g.u.w.RestServiceImpl : Exception : {} 752fae70-3650-4a23-9150-b806a8f54cb6 APP/TASK/3a966d54/0 2021-10-07T06:42:52.688332111Z OUT org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://another-microservice-endpoint": PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target; nested exception is javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

For an example of a sample task, refer to Cloudfoundry Task Demo.

Environment

Product Version: 2.10

Resolution

There is no difference between how certificates are handled when the app is ran as an app or as a task. For more information on this topic, refer to How to tell application containers running Java apps to trust self-signed certifications or a private or internal CA.

However, in example above, the start command differs when the app is ran as an app from the start command used when the app is ran as a task.

The command ".java-buildpack/open_jdk_jre/bin/java org.springframework.boot.loader.JarLauncher" does not load the TrustManager. As a result, it did not have the necessary certs. 

To work around this issue and run your app as a task, follow these steps:

1. Go to your Apps Manager, look for your app-running-as-task, and go to settings.

2. Copy the start command, which looks similar to this:

Start Command: JAVA_OPTS="-agentpath:$PWD/.java-buildpack/open_jdk_jre/bin/jvmkill-1.16.0_RELEASE=printHeapHistogram=1 -Djava.io.tmpdir=$TMPDIR -XX:ActiveProcessorCount=$(nproc) -Djava.ext.dirs=$PWD/.java-buildpack/container_security_provider:$PWD/.java-buildpack/open_jdk_jre/lib/ext -Djava.security.properties=$PWD/.java-buildpack/java_security/java.security $JAVA_OPTS" && CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculator-3.13.0_RELEASE -totMemory=$MEMORY_LIMIT -loadedClasses=15807 -poolType=metaspace -stackThreads=250 -vmOptions="$JAVA_OPTS") && echo JVM Memory Configuration: $CALCULATED_MEMORY && JAVA_OPTS="$JAVA_OPTS $CALCULATED_MEMORY" && MALLOC_ARENA_MAX=2 SERVER_PORT=$PORT eval exec $PWD/.java-buildpack/open_jdk_jre/bin/java $JAVA_OPTS -cp $PWD/. org.springframework.boot.loader.JarLauncher

Use this command to run the task. It should now load the TrustManager, which loads the needed certificates run your app as a task.