CredHub reference can't be interpolated in container due to improper HTTP(S) Proxy Configuration in Tanzu Application Service (TAS) for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

While upgrading from lower releases to the following versions listed below, apps with credhub-ref in VCAP_SERVICES may fail to interpolate during app staging or restart due to an Unable to interpolate error:

• TAS v2.7.18 or above
• TAS v2.8.12 or above
• TAS v2.9.6 or above

Unable to interpolate credhub refs: Unable to interpolate credhub references: Post "https://credhub.service.cf.internal:8844/api/v1/interpolate": Forbidden

This error only occurs when all the conditions below are met: 

• The apps have credhub-ref in VCAP_SERVICES environment variable. For example, Spring Cloud Services client apps.
• The apps are configured with a https_proxy, global environment-variable-group, or an app environment variable. 
• no_proxy is not configured properly with .cf.internal domain. According to Configuring Proxy Settings for All Apps, VMware recommends this value contains the domain .cf.internal, which is used by the platform to communicate with CredHub.

App container reaches out to credhub.service.cf.internal (it's internal host) to interpolate credhub-ref in VCAP_SERVICES during initialization. When https_proxy is configured but the CredHub host is not excluded by no_proxy, the app container will access credhub.service.cf.internal through https_proxy, which is usually being blocked.

Note: There was no problem because the credhub-cli included by Diego didn't support proxy feature, https_proxy configuration was ignored until TAS v2.7.18 /  v2.8.12 / v2.9.6 releases.

Environment

Product Version: 2.7

Resolution

VMware recommends the operator to review your foundations before upgrading to TAS v2.7.18 /  v2.8.12 / v2.9.6 or above. 

If http(s)_proxy is configured globally with staging / running-environment-variable-group, make sure that no_proxy contains .cf.internal globally as well. 

Since an app-specific environment variable can overwrite above global config, for apps with their own no_proxy config, please have app owners update no_proxy to contain .cf.internal.

Below are the steps to list impacted apps:

1.  cf login as admin user. 
2. cf curl /v2/apps | jq .total_pages to get apps total pages. 
3. replace the PAGE in below command with value returned by step 2
4. for page in {1..PAGE}; do cf curl /v2/apps?page=$page | jq '.resources[].metadata.guid' | tr -d \"; done | while IFS= read -r line; do echo $line; cf curl /v2/apps/$line/env | egrep 'credhub-ref|https_proxy|no_proxy'; done

The output would look similar to the following: 

c016c423-d462-4cde-9c31-65ce300d6d6e
6f3ea600-46cd-4e3b-814c-f4b20d7e6174
519a956b-b28d-49c5-bf0c-d7a8eba38b24
      "credhub-ref": "/c/p.spring-cloud-services-scs-mirror-service/8bc8928d-bf15-40a9-b18e-7a65a92b0c3c/credentials"
bf221945-0324-4208-92c7-898726d76692
  "https_proxy": "http://aaa.bbb.ccc.ddd:8080"
      "credhub-ref": "/c/p.spring-cloud-services-scs-service-broker/e9ca123d-c312-4f57-88a4-aef09e887f83/d443c442-c76f-4b7b-b35c-e8227d30958b/credentials-json"

In this example, the app with GUID bf221945-0324-4208-92c7-898726d76692 has both https_proxy and credhub-ref.

It won't be able to interpolate credhub-ref after the upgrade, therefore no_proxy must be configured to contain  .cf.internal before the upgrade.

Please restart the app for the change to take effect.