App using Credhub service, cf unbind-service fails with "Service broker error: The request could not be completed because the permission does not exist or you do not have sufficient authorization."

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

In app using Credhub service, cf unbind-service failed with the following message:

"Service broker error: The request could not be completed because the permission does not exist or you do not have sufficient authorization."

This problem only happens when the application is using a Credhub service. When the customer does a cf unbind-service, it reports the following error:

Error: An unbind operation for the service binding between app <app_name> and service instance <service_instance_name> failed: Service broker error: The request could not be completed because the permission does not exist or you do not have sufficient authorization.

Environment

Product Version: 2.8

Resolution

This has been identified as a bug and is resolved in Service broker version 1.5.0. Refer to the release notes.

When you use cf unbind , the credhub-service-broker will clean up the following:

1. The "service binding actor key" which was used by the broker to associate the binding ID with the app ID.

2. The permission for the app to access the credentials.

This error occurs because one or all of the above items have been deleted already. You can use the  credhub get-permission  and credhub get outputs to verify this.

This error also occurs because when you use cf unbind, the broker returns an error because it fails to clean them up (already gone).

Workaround

1. Get GUIDs of the issued application:

cf app <app> --guid
cf service <service> --guid
cf curl /v2/apps/<app_guid>/service_bindings

Note: The value of resouces.metadata.guid is service_binding_guid.

2. Refer to How to access Tanzu Application Service's (TAS's) CredHub with the CredHub CLI to target the TAS Credhub. Check the credhub-service-broker permission:

credhub get-permission -a uaa-client:credhub-service-broker -p '/credhub-service-broker/*'

The credhub-service-broker has all permissions. The expected output is listed below:

actor: uaa-client:credhub-service-broker
operations:
- read
- write
- delete
- read_acl
- write_acl
path: /credhub-service-broker/*

3. Check for the existence of the service binding actor key:

credhub get -n /credhub-service-broker/credhub/<service_instance_guid>/<service_binding_guid>

If the service binding actor key is missing and reports the following error:

The request could not be completed because the permission does not exist or you do not have sufficient authorization.

Then, you can recreate the service binding actor key:

credhub set -n /credhub-service-broker/credhub/<service_instance_guid>/<service_binding_guid> -v mtls-app:<app_guid> -t value

4. Check the permissions for the app to access the credentials:

credhub get-permission -a mtls-app:<app_guid> -p /credhub-service-broker/credhub/<service_instance_guid>/credentials

The app should only need to read. The expected output is as follows:

actor: mtls-app:<app_guid>
operations:
- read
path: /credhub-service-broker/credhub/<service_instance_guid>/credentials

If the permission for the mtls-app is missing and reports the error as shown below:

The request could not be completed because the permission does not exist or you do not have sufficient authorization.

Then you need to restore the failed app's permission:

credhub set-permission -a mtls-app:<app_guid> -p /credhub-service-broker/credhub/<service_instance_guid>/credentials -o read

 

Take-away

In the next version of the CredHub broker (v1.50), the broker will not return an error to CAPI. It will return code 410 (Already Gone) to CAPI to indicate the clean up already happened. This issue won't happen. But for now, as a workaround, you can try recreating both (step 1) and (step 2) before you cf unbind.