Adding Another Certificate to the Ops Manager Trusted Certificates Fails the UAA Jobs to Deploy

Products

• Operations Manager

Issue/Introduction

Symptoms:

When attempting an add a certificate where there are already existing certificates within the Operations (Ops) Manager > security section, UAA jobs will fail on apply changes.

Error Message:

Deployment failed at UAA update
Started updating instance uaa 
Started updating instance uaa > uaa/4efb3606-b2a8-48fb-a10d-70c654ec71e3 (0) (canary). Failed: Action Failed get_task: Task 16ab6cf5-6c6d-43d9-5c84-76d49197508e result: 1 of 3 pre-start scripts failed. Failed Jobs: uaa. Successful Jobs: route_registrar, consul_agent. (00:00:11)
Error 450001: Action Failed get_task: Task 16ab6cf5-6c6d-43d9-5c84-76d49197508e result: 1 of 3 pre-start scripts failed. Failed Jobs: uaa. Successful Jobs: route_registrar, consul_agent.

Within the uaa logs pre-start.stdout.log you will see:

A failed run looks like the following (note lack of java certificates being processed):

uaa-pre-start - completed at Fri Feb 24 19:56:43 UTC 2017
uaa-pre-start - starting at Wed Mar 15 19:49:57 UTC 2017
uaa-pre-start - starting at Wed Mar 15 20:03:38 UTC 2017

Note of uaa-pre-start script starting, but no importing of certificates.

A successful run looks like this:

uaa-pre-start - starting at Fri Feb 24 19:55:48 UTC 2017
Processing certificates for Java cacerts file
Processed certificate 1 of 175
etc.. 

 

Environment

Cause

This appears to be a bug introduced here where set -e was added to the uaa pre-start scripts

https://github.com/cloudfoundry/uaa-release/commit/9777d932e9ec89d1eb5c09fa1cff0501dcbb215f

We modified uaa pre-start script with set -ex to see where it fails and because diff returns exit code 1 given there are differences and set -e is enabled the bash shell will exit pre-start and not log any errors. Diff always returns a return of 1 on mismatched results, causing the script to fall through to exit when adding certificates.

uaa/4efb3606-b2a8-48fb-a10d-70c654ec71e3:~$ /var/vcap/jobs/uaa/bin/pre-start
++ date
+ echo 'uaa-pre-start - starting at Fri Mar 17 15:01:52 UTC 2017'
uaa-pre-start - starting at Fri Mar 17 15:01:52 UTC 2017
+ CERT_FILE=/etc/ssl/certs/ca-certificates.crt
+ CONF_DIR=/var/vcap/jobs/uaa/config
+ CACHE_DIR=/var/vcap/data/uaa/cert-cache
+ mkdir -p /var/vcap/data/uaa/cert-cache
+ CERT_CACHE_FILE=/var/vcap/data/uaa/cert-cache/cacerts-cache.txt
+ TRUST_STORE_FILE=/var/vcap/data/uaa/cert-cache/cacerts
++ grep 'END CERTIFICATE' /etc/ssl/certs/ca-certificates.crt
++ wc -l
+ CERTS=177
+ '[' -a /var/vcap/data/uaa/cert-cache/cacerts-cache.txt ']'
+ '[' -a /var/vcap/data/uaa/cert-cache/cacerts ']'
+ diff /var/vcap/data/uaa/cert-cache/cacerts-cache.txt /etc/ssl/certs/ca-certificates.crt

Resolution

1. BOSH SSH into each UAA vm
2. On each UAA VM, sudo mv /var/vcap/data/uaa/cert-cache/cacerts-cache.txt /root
3. Apply Changes via Ops Manager.

Permanent Fix

Upgrade to Elastic Runtime 1.9.13 or later (UAA 24.6)