PASW Diego cell fails to fetch package from BOSH director internal blobstore with error "certificate signed by unknown authority"

Products

• Operations Manager

Issue/Introduction

If a customer meets the following conditions, they may hit the error:

1. At TAS 2.7, BOSH director uses the internal blobstore and disables TLS. Config path: BOSH director tile -> Director Config -> Blobstore Location -> Internal -> Enable TLS (checkbox). See the Ops Manager 2.7 installation doc.

2. In addition, the Include Ops Manager Root CA in Trusted Certs checkbox is disabled.

3. Customer has upgraded Ops Manager/BOSH director to v2.9+. They could hit the following error when deploying the tile that uses am stemcell containing BOSH agent v2.261.0+. For example, a customer's TAS Windows tile v2.7 uses stemcell 2019.26 and hits the error as follows:

Task 593170 | 04:52:56 | Compiling packages: bosh-dns-windows/a7f5d5ebe54f0323c8135615e1725c042d1ce4d3e271d502080d767b6d2e3a65 (00:06:08)
Updating deployment:
  Expected task '593170' to succeed but state is 'error'
Task 593170 | 04:59:39 | Updating instance windows_diego_cell: windows_diego_cell/bed47e81-4f99-4ea7-b548-c35e200e528b (0) (canary) (00:00:02)
                      L Error: Action Failed get_task: Task 47c4f09f-e2bd-4256-41d8-12a4944c3389 result: Preparing apply spec: Preparing package bosh-dns-windows: Fetching package blob: Excuting GET request: Get https://10.209.10.11:25250/signed/0b/05c0db2e-6c08-4ac5-8ac3-f0920f7c9a0b?e=86400&st=qqMYE87aOrhMu9eGg_-EqDsFFSaMG2dmuKFrKg-cTJk&ts=1607662780: x509: certificate signed by unknown authority

Root Cause:

1. Since Ops Manager/BOSH director version 2.8, TLS is always enabled for BOSH director internal blobstore (see release note).

2. Since Ops Manager/BOSH director version 2.9, the Signed URLs feature is always enabled. This feature is supported by Windows stemcell 2019.17+ (see release note). It means the bosh agent running in the Windows Diego cell would manage artifacts on the BOSH internal blobstore using signed URLs rather than blobstore credentials.

3. After upgrading OpsManager/BOSH director to version 2.9+, the BOSH agent in the Windows cell does not know the BOSH internal blobstore CA (which is indeed the Ops Manager root CA if the customer root CA is not used) and would hit the mentioned error when handling the signed URL to fetch packages from BOSH internal blobstore. The bosh agent v2.261.0+ is not able to fall back to username/password authentication because of this change.

Environment

Product Version: 2.9

Resolution

After upgrading Ops Manager/BOSH director to version 2.9+, the BOSH internal blobstore CA has already been populated into the BOSH director manifest (maintained by Ops Manager) and has been also populated into the configuration of the director job (location: /var/vcap/jobs/director/config/director.yml in BOSH director VM). To verify this, you could ssh to the BOSH director VM and verify that the configuration file contains the ca (see example below). 

bosh/0:/var/vcap/jobs/director/config# cat /var/vcap/jobs/director/config/director.yml | jq '.agent.env.bosh.blobstores'
[
  {
    "options": {
      "endpoint": "https://10.213.38.218:25250",
      "password": "Ua8zoYjPLu8k-FNxb8vih4cZ3ir-lhSV",
      "tls": {
        "cert": {
          "ca": "-----BEGIN CERTIFICATE-----....I3SYskMvc4Y4++1bPSA....\n-----END CERTIFICATE-----\n"
        }
      },
      "user": "blobstore"
    },
    "provider": "dav"
  }
]

Therefore, the workaround is to push the CA into the BOSH agent's configuration in the tile hitting the error. For example, the BOSH agent's configuration file location in Windows cell is at c:\var\vcap\bosh\settings.json. The steps are as follows:
 

• Select checkbox Recreate VMs deployed by the BOSH Director in Director Config
• Apply Change selectively with the tile hitting the error.