Credhub Bindings Ordering Change in VCAP_SERVICES

Products

• VMware Tanzu Application Service

Issue/Introduction

Customers may notice that when trying to access the credhub data in VCAP_SERVICES, the ordering has changed. Credhub bindings are ordered by instance GUID, which may break their property interpolation.

${vcap:services:credhub:2:credentials:keystore} (the binding at index 2 changed, and the expected key isn’t present)

This may result in issues with application property interpolation, as the binding in index2 was changed, and the expected key was missing.

Environment

Effected versions

 

This effects https://github.com/cloudfoundry/capi-release/releases/tag/1.218.0 and above. This means it is present in the following TAS versions:

• 6.0.22
• 6.0.21
• 10.2.5
• 10.2.4

It would appear as though we relied on implicit ordering (by ID) and that has changed, while not obvious, it appears to be due to the use of the partition and refactoring to support binding rotation in the future.

Cause

Root Cause Analysis

This appears to have been introduced in open source as part of this commit by SAP to support binding rotation: https://github.com/cloudfoundry/cloud_controller_ng/commit/79b4da307661ec2a03830731414fa2c3a70d94f9#diff-403a1ff3052c3335b1d4f0db3569c895bf7ba4ca01539c562d009394c907d64bR148  and this migration https://github.com/cloudfoundry/cloud_controller_ng/commit/f01165168799634a2e90cc923db78bff1d40d542 . I believe the introduction of the index:

 

    add_index %i[app_guid service_instance_guid],                    name: :service_bindings_app_guid_service_instance_guid_index 

This means that the query now uses this index, and it utilizes the natural order of that index, which is by service_instance_guid, rather than the previously used natural order by id. 

Resolution

Workaround

Applications when consuming bindings should not rely on the indexed order of the bindings in VCAP services, instead, they should rely on fetching the credential by some information in the binding itself, e.g. instance name.

Customers will need to find something in the binding to distinguish each credential. 

Assuming a java app and using https://github.com/pivotal-cf/java-cfenv you can fetch all bindings for a service with something like this:

 

CfEnv cfEnv = new CfEnv();List<CfService> credhubServices = cfEnv.findServicesByName("credhub");CfService myService1 = credhubServices.stream().filter(s -> s.getName().equals("my-instance-name-1")).findFirst().get()CfService myService2 = credhubServices.stream().filter(s -> s.getName().equals("my-instance-name-2")).findFirst().get()

 

Fixing 

 

Fix has been introduced in 6.0.23, 10.2.6, and 10.3.2.