cf unbind-service fails with "permission does not exist or you do not have sufficient authorization."

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

cf unbind-service fails with the following error:

Unbinding app <app_name> from service <service_name> in org <org> / space <space> as <[email protected]>
An unbind operation for the service binding between app <app_name> and service instance <service_instance> failed: Service broker error: The request could not be completed because the permission does not exist or you do not have sufficient authorization.
FAILED

Service broker logs reports the following errors:

a7d5eb9e-#######-4fe6-#####-9fe1e03527fa APP/PROC/WEB/0 2021-03-31T01:54:11.477321598Z OUT {"timestamp":"1617155651.477131844","source":"secure-credentials-broker","message":"secure-credentials-broker.retrieving service binding actor for key /credhub-service-broker/credhub/######-a6d7-45b6-bcbd-###########/1441bec8-21f5-4378-bca8-#########","log_level":1,"data":{}}
a7d5eb9e-#####-4fe6-#####-9fe1e03527fa APP/PROC/WEB/0 2021-03-31T01:54:11.534857088Z OUT {"timestamp":"1617155651.534660578","source":"secure-credentials-broker","message":"secure-credentials-broker.unbind.unknown-error","log_level":2,"data":{"binding-id":"1441bec8-####-4378-####-32da7b094a42","correlation-id":"######-171a-4f0d-#####-62c339cf56be","error":"The request could not be completed because the permission does not exist or you do not have sufficient authorization.","instance-id":"2a4a0cdd-######45b6-###-#######","session":"7810"}}

 

From the CredHub service broker logs, you see that the last thing that happens is an attempt to retrieve service binding actor for key "/credhub-service-broker/credhub/2a4a0cdd-###-45b6-bcbd-######5c5b396e/1441bec8-####-4378-#####-32da7b094a42"  before it reports the error.

This should return the actor mtls-app:<APP_GUID> for the broker to then delete that permission. We expect that the /credhub-service-broker/credhub/2a4a0cdd-####-45b6-#####-ebfc5c5b396e/######-21f5-4378-bca8-32#######a42 key is the one that is inaccessible to the service broker or deleted. The service broker should have access to that key, because it has /credhub-service-broker/* so if you are encountering this error then this key was likely deleted somehow. If it is not present, it can be recreated with the value set to the expected actor mtls-app:<APP_GUID>.

Environment

Product Version: 2.10

Resolution

Perform the following steps to resolve this issue:

1. In the service-broker logs, check for an error which states "retrieving service binding actor for key":

"message":"secure-credentials-broker.retrieving service binding actor for key /credhub-service-broker/credhub/2a4a0cdd-####-45b6-####-ebfc5c5b396e/1441bec8-#####-####-#####-32da####94a42","log_level":1,"data":{}} a7d5eb9e-2c24-4fe6-####-9fe1e03527fa APP/PROC/WEB

2. Connect to TAS Credhub CLI by following the steps in this article:
How to access TAS's CredHub with the CredHub CLI

3. Attempt to "get" the key that failed retrieval in the service broker logs (Step 1): 

credhub get -n /credhub-service-broker/credhub/2a4a0cdd-####-45b6-####-ebfc5c#####/1441bec8-21f5-####-bca8-32d######2

4. If retrieval fails, then attempt to set the key value to expected an actor (mtls-app:<APP_GUID>) by using the following command:

credhub set -n /credhub-service-broker/credhub/<service_instance_guid>/<service_binding_guid> -v mtls-app:<APP_GUID> -t value

Where <APP_GUID> is the GUID of the application that you are trying to unbind service from (cf app <app-name --guid).

5. Re-test unbinding the service from your application and confirm if it now succeeds.