Trusted CA certificates missing in docker apps under "/usr/local/share/ca-certificates"

Products

• VMware Tanzu Application Service

Issue/Introduction

Credhub interpolation fails when executing 'https://credhub.service.cf.internal:8844/api/v1/interpolate' from a docker based image app in TAS 

Environment

TAS - Tanzu Platform for Cloud Foundry

4.x
5.x
6.x

Cause

Both docker images and buildpack based containers install system certificates to /etc/cf-system-certificates. 

This is the path that all apps (docker and buildpack) should rely on to find certs as documented (https://docs.vmware.com/en/VMware-Tanzu-Application-Service/6.0/tas-for-vms/deploy-apps-trusted-system-certificates.html)

It happens that buildpack based apps also install them to /usr/local/share/ca-certificates/. 
Buildpack apps are based on cflinuxfs3 or cflinuxfs4 root filesystems provided by platform and they install ca-certificates package to base ubuntu OS which creates /usr/local/share/ca-certificates/, they split system certificates into separate files and saves them in /usr/local/share/ca-certificates and then run update-ca-certificates which creates symlinks in /etc/ssl/certs. This is to facilitate software that is installed on cflinuxfs3/cflinuxfs4 like openssl. This is very specific to the operating system they are based on (Ubuntu) and software they run there (e.g. openssl).

When app is pushed with docker image this docker image can be based on any OS and run any software which might not respect /usr/local/share/ca-certificates/ so Diego does not set this up in docker based apps. Instead it is using generic conventional path /etc/cf-system-certificates where it puts system certificate and then it would be the responsibility of the docker image or an app running in them to do whatever whey need to do with it. For apps this path is available in environment variable CF_SYSTEM_CERT_PATH

Resolution

As workaround add to the ENTRYPOINT of the dockerfile

cp /etc/cf-system-certificates/* /usr/local/share/ca-certificates/
/usr/sbin/update-ca-certificates