App keeps crashing due to error "Unable to interpolate credhub refs ... because the credential does not exist or you do not have sufficient authorization."

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

An app on the Tanzu Application Service platform keeps crashing / restarting with the following error message:

Unable to interpolate credhub refs ... The request could not be completed because the credential does not exist or you do not have sufficient authorization

Once the app is restarted from Apps Manager or via cf cli, the error disappears and it starts successfully.

Environment

Product Version: 2.7

Resolution

When binding an app with services which support CredHub interpolation, in VCAP_SERVICES of `cf env` output, the binding credentials would be a CredHub reference instead of plain text credentials. Here is an example an app binding to a Spring Cloud Services config-server:

{
 "VCAP_SERVICES": {
  "p.config-server": [
   {
    "binding_name": null,
    "credentials": {
     "credhub-ref": "/c/p.spring-cloud-services-scs-service-broker/e9ca123d-c312-4f57-88a4-aef09e887f83/cf4819b2-d3b9-405b-8df9-37649454a4a1/credentials-json"
    },
    "instance_name": "config-server",
   ...
   }
  ]
 }
}

With the CredHub reference, during container initialization, the Diego component reaches out to CredHub VMs in TAS to interpolate the credential. If that CredHub reference doesn't exist on CredHub VMs for any reasons, the error below will be reported in app log:

2021-02-19T14:14:04.61+0100 [APP/PROC/WEB/0] OUT Exit status 3
2021-02-19T14:14:04.61+0100 [APP/PROC/WEB/0] ERR Unable to interpolate credhub refs: Unable to interpolate credhub references: The request could not be completed because the credential does not exist or you do not have sufficient authorization.

This usually occurs with the following scenario:
 

1. A developer unbinds/rebinds an app from/with a service. At this point, cloud controller DB gets updated with a new CredHub reference, old CredHub references get removed from CredHub.  
2. The developer doesn't restart the app, thus the new CredHub reference is not updated into Diego BBS. 
3. The app gets restarted by Diego instead of cloud controller, for example during platform update/upgrade or after an app crash. 
4. During the container initialization, it fails to interpolate the CredHub reference because the old one has already been removed from CredHub VM.
5. This keeps repeating until restarting the app from Apps Manager or via cf cli, which triggers the CredHub reference update from cloud controller to Diego BBS. 

To avoid such a scenario, it's recommended to restart or restage the app after any service binding or service key operations.