The errand "rotate_cc_database_key" fails with the error "exit status 1"

Products

• VMware Tanzu Application Service

Issue/Introduction

When enable the errand “Rotate CC Database Key” during “Apply Changes”, this errand might fail with the error “exit status 1”.

Errand 'rotate_cc_database_key' completed with error (exit code 1)
Exit code 1

Cause

When reviewing the stack trace, you might notice the following lines on top. The messages report an API error in cloud controller, which suggests the encryption key is not found. 

Stderr     rake aborted!  
           CloudController::Errors::ApiError: Please set the desired encryption key in the manifest at ‘cc.database_encryption.current_key_label’ (CloudController::Errors::ApiError)  
           /var/vcap/data/packages/cloud_controller_ng/a5219efa3aeca8394c2ded2c7b0fc7000656779d/cloud_controller_ng/lib/cloud_controller/errands/rotate_database_key.rb:21:in `no_encryption_key!'  
           /var/vcap/data/packages/cloud_controller_ng/a5219efa3aeca8394c2ded2c7b0fc7000656779d/cloud_controller_ng/lib/cloud_controller/errands/rotate_database_key.rb:7:in `perform'

Resolution

 The Cloud Controller has its own database, the CCDB, in which it stores information about objects in TAS for VMs such as apps. The CCDB encryption key is used to encrypt sensitive data at rest in the CCDB such as app environment variables. The “Rotate CC Database Key” errand retrieves the encryption key from the manifest file and applies it to the CCDB. It is not enough to run the errand itself, and it is recommended following the Rotating the Cloud Controller database encryption key to rotate.

In terms of this error itself, we can fix it by adding the encryption key in the Ops Manager UI --> TAS Tile --> Cloud Controller --> the Encryption key ledger field, followed by “Rotate CC Database Key” errand again.