The config-server reports error "Unable to invoke Cipher due to bad padding" when attempting to decrypt encrypted property

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

The backend app of config-server service reports error "Unable to invoke Cipher due to bad padding" when it tries to decrypt some encrypted property which is prefixed with the string {cipher}.  The error messages look like the following.

2024-05-17T12:51:23.83+0530 [APP/PROC/WEB/1] OUT ESC[35m[http-nio-8080-exec-5]ESC[0;39m ESC[34mINFO ESC[0;39m o.s.c.c.s.e.NativeEnvironmentRepository.clean - Adding property source: Config resource 'file [/home/vcap/app/config-repo-default/prod/config/application.yml]' via location 'file:/home/vcap/app/config-repo-default/prod/config/'
2024-05-17T12:51:23.83+0530 [APP/PROC/WEB/1] OUT ESC[35m[http-nio-8080-exec-5]ESC[0;39m ESC[31mWARN ESC[0;39m o.s.c.c.s.e.CipherEnvironmentEncryptor.decrypt - Cannot decrypt key: Protector:AuthCredentials:Password (class java.lang.IllegalStateException: Unable to invoke Cipher due to bad padding)

 

Environment

• TAS 3.x, 4.x
• SCS 3.x

Cause

The error "Unable to invoke Cipher due to bad padding" usually means config-server is unable to decrypt the encrypted data with the configured encrypt key. Some typical scenarios:

1. The encrypt key has been changed but the encrypted data is still associated with old encrypt key
2. The encrypted data is updated in the source directly. However the updated data is actually encrypted with some key other than the one configured for the config-server

Resolution

• If you know what encrypt key is set for the config-server and it's correct, try to encrypt the data again using the config-server /encrypt endpoint. Then update the data source with new encrypted data. 
• If the encrypt key has been mistakenly changed, use cf update-service command to update encrypt key to the one which was previously used to encrypt data. You can also utilise the /encrypt endpoint to encrypt data again after the key is updated
• The config-server URL is the dashboard URL shown by cf service <config-server-service-name> command
• Refer to SCS document for more details

Additional Information

How to retrieve the encrypt key configured for one config-server service

1. Log into TAS credhub by referring to "Scenario 2" in this KB article

2. Run CF CLI to get the path in Credhub storing the encrypt key 

• cf service <config-server service name> --guid  # to retrieve service instance guid
• cf target -o p-spring-cloud-services -s <config-server service instance guid>
• cf env config-server  # note down the value of credhub-ref key in VCAP_SERVICES section. For example

$ cf env config-server
Getting env variables for app config-server in org p-spring-cloud-services / space 915cf0e9-1a20-####-####-e20d0e984929 as admin...
System-Provided:
VCAP_SERVICES: {
  "p.mirror-service": [
    {
      "binding_guid": "0b0fa647-7e3f-####-####-c7b01edc3482",
      "binding_name": null,
      "credentials": {
        "credhub-ref": "/c/p.spring-cloud-services-scs-mirror-service/2b3557d1-b06a-####-####-560971685709/credentials"
      },
      "instance_guid": "2b3557d1-b06a-####-####-560971685709",
      "instance_name": "mirror-svc",
      "label": "p.mirror-service",
      "name": "mirror-svc",
      "plan": "standard",
      "provider": null,
      "syslog_drain_url": null,
      "tags": [],
      "volume_mounts": []
    }
  ]
}

3. Run command credhub get -n <credhub-ref value retrieved in preceding step>. For example,

$ credhub get -n /c/p.spring-cloud-services-scs-mirror-service/2b3557d1-b06a-####-####-560971685709/credentials