Credhub post-start script fails due to encryption key error

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Apply changes is failing with error reporting post-start script failed for credhub:

Updating deployment: Expected task '3022962' to succeed but state is 'error' Exit code 1 Task 3022962 | 18:10:24 | Updating instance credhub: credhub/47487341-a1b6-4429-82d6-b48aa7e474ae (0) (canary) (00:11:37) L Error: Action Failed get_task: Task c5bc54c0-c02e-49dc-67ad-bdcadc74e906 result: 1 of 2 post-start scripts failed. Failed Jobs: credhub. Successful Jobs: bosh-dns. Task 3022962 | 18:10:24 | Error: Action Failed get_task: Task c5bc54c0-c02e-49dc-67ad-bdcadc74e906 result: 1 of 2 post-start scripts failed. Failed Jobs: credhub. 

/var/vcap/sys/log/credhub/credhub-post-start.log will show error retrying health-check of credhub job.

/var/vcap/sys/log/credhub/credhub.log reports error "The encryption keys provided cannot decrypt any of the 578 value(s) in the database."
 

2021-01-11T19:09:24.138Z [main] .... ERROR --- SpringApplication: Application run failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'asyncEncryptionKeyRotationScheduler' defined in URL [jar:file:/var/vcap/data/packages/credhub/2caa3b30f63253f19e9a505f7d3f699360ef6239/credhub.jar!/BOOT-INF/lib/encryption.jar!/org/cloudfoundry/credhub/services/AsyncEncryptionKeyRotationScheduler.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.cloudfoundry.credhub.services.AsyncEncryptionKeyRotationScheduler]: Constructor threw exception; nested exception is java.lang.RuntimeException: The encryption keys provided cannot decrypt any of the 578 value(s) in the database. Please make sure you've provided the necessary encryption keys.

Environment

Product Version: 2.8

Resolution

The error "The encryption keys provided cannot decrypt any of the 578 value(s) in the database." means that encryption key configured for Credhub is unable to decrypt values in database.

This likely indicates that the Credhub encryption key has changed at some point. You can run the following commands to check encryption key history (see Credhub login KB): 

credhub api https://<director IP>:8844/api/ --skip-tls-validation

credhub login --client-name=director_to_credhub --client-secret=<secret>

credhub find -p / | grep credhub_internal_provider_keys

credhub get -n /opsmgr/cf-f18558e6a9d2518299f4/credhub_internal_provider_keys/0/key --versions=10 

The last command above will list out all the prior versions of encryption key.  For the safest resolution to this error, we would recommend adding all these versions to the TAS tile as encryption keys (only check off active for the key you would like to use going forward). You can do this by navigating to OpsManager and adding further keys under:

TAS > Credhub >  Internal encryption provider keys  

These steps will guarantee that Credhub can decrypt all the values in database. For more information, see Credhub configuration document.