Concourse deployment fails to start a job with error "certificate signed by unknown authority"

Products

• Operations Manager

Issue/Introduction

Symptoms:
BOSH job fails to generate certificate and cannot start the job.
Steps to find the error:

• BOSH SSH <instance>
• sudo -i
• less /var/vcap/sys/log/<job-name>/<job_name>.stderr.log

time="2018-09-13T15:28:56Z" level=error msg="Error creating BOSH Client: Fetching info: /
Performing request GET 'https://192.168.1.2:25555/info': Performing GET request: /
Retry: Get https://192.168.1.2:25555/info: x509: certificate signed by unknown authority" source="<job_name>.go:264"

This is a certificate error while fetching info from the Director VM.

Environment

Cause

The pipelines often use om command-line utility to communicate with Ops Manager. It is a common to see the arguments formatted in certain ways. However, newer versions of om command-line utility comes with a limitation.

Recent changes in om command-line utility in version 0.39.0 and above causes this issue.
The change is listed in the Release Notes:

• Add --format to specific commands that use it and remove global --format flag (breaking change)

The om command below fails silently:

om \
--target "https://${OPSMAN_DOMAIN_OR_IP_ADDRESS}" \
--skip-ssl-validation \
--username "$OPSMAN_USERNAME" \
--password "$OPSMAN_PASSWORD" \
--format=json certificate-authorities | jq -r '.[0].cert_pem' > pcf_bosh_ca_cert.pem

If this command is run with --format flag anywhere but in the end, we see the error below:

flag provided but not defined: -format

This indicates that the flag -format is not accepted globally.

Resolution

Make sure the om command runs with correct syntax.
In this case, the flag -format must be used in the end as it is not accepted globally:

om \
--target "https://${OPSMAN_DOMAIN_OR_IP_ADDRESS}" \
--skip-ssl-validation \
--username "$OPSMAN_USERNAME" \
--password "$OPSMAN_PASSWORD" \
certificate-authorities --format=json | jq -r '.[0].cert_pem' > pcf_bosh_ca_cert.pem