How to rotate expired BOSH DNS CA certificate

Products

• Operations Manager

Issue/Introduction

BOSH DNS CA/leaf certificate can be automatically rotated by enabling "Enable automatic rotation of the BOSH DNS CA certificate" in Director Config. The rotated BOSH DNS certificates can be deployed to VMs during the Apply Changes action when new stemcells are also being deployed to those VMs. Please refer product document for details. 

In the case that above feature is not enabled and the certificate is not rotated manually before expiration, this article will describe how to rotate BOSH DNS CA certificate when it already expired. 

Resolution

Before BOSH DNS CA expiration, it could be manually - Rotate a single CA and its leaf certificates with Credhub maestro. However in the case CA has already expired, step 1~4 have to be merged as: 

1. maestro regenerate ca --name "/opsmgr/bosh_dns/tls_ca"
2. maestro update-transitional signing --name "/opsmgr/bosh_dns/tls_ca"
3. maestro regenerate leaf --signed-by "/opsmgr/bosh_dns/tls_ca"
4. Kick off "Apply Change", for service tiles please turn on "Upgrade all service instances"

Above deployment may fail with tile or service instance deployments, because some drain scripts relies on BOSH-DNS, but it already stops functioning. In this case, please rerun the deployment manually with --skip-drain. 

1. `bosh -d FAILED_DEPLOYMENT manifest > deployment.yaml`
2. `bosh -d FAILED_DEPLOYMENT deploy --skip-drain deployment.yaml`

Repeat above steps until new BOSH DNS CA and leaf certificates are deployed on all instances. 

Finally, clean up old expired BOSH DNS CA

1. maestro update-transitional remove --name "/opsmgr/bosh_dns/tls_ca"
2. Kick off "Apply Change", for service tiles please turn on "Upgrade all service instances"