How to recover the platform after the Cloud Controller DB Encryption Key gets updated or changed

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

If the wrong key is configured in the Cloud Controller DB Encryption Key field and "Apply Changes" is executed, commands such as cf push and cf apps may fail. In this instance, requests to the CC API hit a "500" error.

######   Error from cf CLI:   ######
          Staging app and tracing logs...  
          Unexpected Response  
          Response code: 500  
          CC code:       0  
          CC error code:   
          Request ID:    c1790125-9b32-42ac-5750-488d8476ded6::ea743dae-88c1-405f-ab77-83f86fabbd31  
          Description:   {  
            "error_code": "UnknownError",  
            "description": "An unknown error occurred.",  
            "code": 10001

######    Error from cloud_controller_ng log:    ######
{"timestamp":1558469164.8900797,"message":"Request failed: 500: {\"error_code\"=>\"UnknownError\", \"description\"=>\"An unknown error occurred.\", \"code\"=>10001, \"test_mode_info\"=>{
\"description\"=>\"bad decrypt\", \"error_code\"=>\"CF-CipherError\",...

Environment

Cause

Data encrypted with the original encryption key in the CC database cannot be decrypted successfully with the wrong key.

These data includes environment_variables of apps, credentials of service instances, credentials of service bindings, buildpack_url of buildpacks, and auth_password of service broker.

Resolution

Scenario I:

The TAS version is at v2.6 or lower. Follow the guidance as follows.
 

1. First verify if the CC database encryption key property changed in any recent VMware Tanzu Application Service (TAS) deployment change logs. An example is shown below:

Using deployment 'cf-e77ffd17b89e686d3625'
 instance_groups:
 - name: cloud_controller
   jobs:
   - name: cloud_controller_ng
     properties:
       cc:
-         db_encryption_key: "<redacted>"
+         db_encryption_key: "<redacted>" 

2. Check the recent backup of Operations Manager (Ops Manager) in installation.yml to find the original key. Revert to using the original key.

• Case A: If cloud_controller.encrypt_key.value is null (not configured) in the backup installation.yml file, set the Cloud Controller DB Encryption Key field as blank. Save and redeploy TAS.
• Case B: If cloud_controller.encrypt_key.value is not null in backup installation.yml, set Cloud Controller DB Encryption Key field with the original key value. Save and redeploy TAS.

  ###### .  Below is an example in which the value of encrypt_key is configured  ######
   - guid: cloud_controller-e38f3a72fba4901ecadc
      installation_name: cloud_controller
      internet_connected: false
      properties:
      - deployed: true
        identifier: db_encryption_credentials
        value:
          identity: db_encryption
          password: xp_0D#####################LWFfz
      - deployed: true
        identifier: encrypt_key
        value:
          secret: thKpH#################qDQc44

3. [Optional] Ops Manager stores the key in the BOSH director within CredHub. You can verify the change history of these value through the CredHub CLI. Refer to the following KB article and commands:

•  https://knowledge.broadcom.com/external/article?articleId=293458 
• credhub find -n encrypt_key
• credhub get -n /opsmgr/cf-****/cloud_controller/encrypt_key --versions=8   ### display recent 8 changes with the option --versions

4. If you deploy apps, creates service instances, or creates service bindings when the wrong key is in use, you need to clean them up after reverting back the original key. This procedure originates from: https://github.com/cloudfoundry/cloud_controller_ng/wiki/Ruby-Console-Script-to-Find-Fields-that-Cannot-Be-Decrypted.

a. SSH to cloud_controller
b. Access cloud controller ruby console with command: /var/vcap/jobs/cloud_controller_ng/bin/console
c. In the ruby console, run the below script to detect apps and service_instances that need to be deleted.

Encryptor.encrypted_classes.each do |class_name|
  klass = class_name.constantize
  klass.all do |model_instance|
    encrypted_field = klass.all_encrypted_fields.first[:field_name]
    begin
      model_instance.send(encrypted_field)
    rescue => e
      puts "#{e} occurred"
      puts "failed to decrypt #{encrypted_field} for #{klass} with guid #{model_instance.guid}"
    end
  end
end

Below is an example output in which two apps have been detected:

bad decrypt occurred
failed to decrypt environment_variables for VCAP::CloudController::AppModel with guid bd996a57-82e2-430d-96f4-8ab06cd9bf98
bad decrypt occurred
failed to decrypt environment_variables for VCAP::CloudController::AppModel with guid f2a8cdd4-8cba-4f77-8a31-77ad4b1d2844

• Delete the apps with the CC API using the following commands:
  • cf curl /v3/apps/<app-guid>   ###   review app info before delete it
  • cf curl -X DELETE /v3/apps/<app-guid>

Scenario II:

The TAS version is at v2.7 or higher, but you have never rotated Cloud Controller DB encryption key via this procedure. 

In this case, Cloud Controller is still using the legacy encryption key as in earlier TAS versions. Therefore please follow the same guidance as in Scenario I.

Scenario III:

The TAS version is at v2.7 or higher, and you had successfully rotated Cloud Controller DB encryption key via this procedure.