SCS for VMware Tanzu 3.1.3 and SCS for VMware Tanzu 3.1.4 broker and mirror service unable to import root certificates into truststore

Products

• Support Only for Spring

Issue/Introduction

When rotating your foundation's root CAs on Spring Cloud Services (SCS) for VMware Tanzu 3.1.3 and SCS for VMware Tanzu 3.1.4, it fails if there are multiple CAs with the same alias.

Specifically the job's mirror-service and scs-broker are unable to start due to a truststore certificate issue. The following errors are observed: 

 spring-cloud-services/225ef535-cee5-49f3-90f1-0a1b03abd488:/var/vcap/sys/log/truststore-builder/pre-start.stdout.log ... For authorized use only/CN=thawte Primary Root CA - G3/ad3493 to truststore Added /CN=rootCA/59f3c1 to truststore Trusting certs from /var/vcap/jobs/truststore-builder/config/credhub_ca.crt... Added /C=US/O=Pivotal/59f3c1 to truststore Unable to add /C=US/O=Pivotal/59f3c1 to truststore - skipping

Resolution

The preferred resolution is to upgrade to SCS for VMware Tanzu 3.1.5. This resolves the issue with a code fix.

If you cannot upgrade, you can apply one of the following workarounds. The specific workaround is different for SCS for VMware Tanzu 3.1.3 and SCS for VMware Tanzu 3.1.4 - the differences are highlighted.

In the SCS for VMware Tanzu VM:

The script located at /var/vcap/jobs/truststore-builder/bin/pre-start adds CAs to the trust store.

If the alias is not unique it will skip it, showing an error like the one below:

Unable to add /C=US/O=Pivotal to truststore - skipping 

Therefore each CA needs to have a unique alias.

As soon as you can SSH into the SCS for VMware Tanzu machine on an Apply Changes, go to the /var/vcap/jobs/truststore-builder/bin/pre-start script.

Fix for 3.1.3

Insert this into the script between the alias assignment and the keytool to check and make unique:

if [[ $alias == *Pivotal* ]]; then alias+="$(date | md5sum | cut -c1-6)" fi 

In total it will look like this:

trust_cert() {
  local input_file alias
  input_file=$1

  alias="$(openssl x509 -in "$input_file" -noout -subject | sed -e 's/subject= //')"

  if [[ $alias == *Pivotal* ]]; then
    alias+="$(date | md5sum | cut -c1-6)"
  fi

  keytool \
    -importcert \
    -alias "$alias" \
    -file "$input_file" \
    -storepass "$TRUST_STORE_PASSWORD" \
    -keystore "$TRUST_STORE_LOCATION" \
    -trustcacerts \
    -noprompt > /dev/null 2>&1 \
    && echo "Added $alias to truststore" \
    || echo "Unable to add $alias to truststore - skipping"
 } 

Fix for 3.1.4

This line attempts to generate unique aliases on first assignment:

alias="$(openssl x509 -in "$input_file" -noout -subject | sed -e 's/subject= //')/$(date | md5sum | cut -c1-6)" 

The problem is:

date | md5sum | cut -c1-6

If performed rapidly, it yields the same result. Therefore, the result is not unique and skipped, and the deployment fails.

Change to:

alias="$(openssl x509 -in "$input_file" -noout -subject | sed -e 's/subject= //')/$(RANDOM)"

this utilizes linux's RANDOM env variable to generate a random number and therefore each alias will be unique and deployment is successful.