While upgrading from lower releases to the following versions listed below, apps with credhub-ref in VCAP_SERVICES may fail to interpolate during app staging or restart due to an Unable to interpolate error:
Unable to interpolate credhub refs: Unable to interpolate credhub references: Post "https://credhub.service.cf.internal:8844/api/v1/interpolate": Forbidden
This error only occurs when all the conditions below are met:
App container reaches out to credhub.service.cf.internal (it's internal host) to interpolate credhub-ref in VCAP_SERVICES during initialization. When https_proxy is configured but the CredHub host is not excluded by no_proxy, the app container will access credhub.service.cf.internal through https_proxy, which is usually being blocked.
Note: There was no problem because the credhub-cli included by Diego didn't support proxy feature, https_proxy configuration was ignored until TAS v2.7.18 / v2.8.12 / v2.9.6 releases.
VMware recommends the operator to review your foundations before upgrading to TAS v2.7.18 / v2.8.12 / v2.9.6 or above.
If http(s)_proxy is configured globally with staging / running-environment-variable-group, make sure that no_proxy contains .cf.internal globally as well.
Since an app-specific environment variable can overwrite above global config, for apps with their own no_proxy config, please have app owners update no_proxy to contain .cf.internal.
Below are the steps to list impacted apps:
The output would look similar to the following:
c016c423-d462-4cde-9c31-65ce300d6d6e 6f3ea600-46cd-4e3b-814c-f4b20d7e6174 519a956b-b28d-49c5-bf0c-d7a8eba38b24 "credhub-ref": "/c/p.spring-cloud-services-scs-mirror-service/8bc8928d-bf15-40a9-b18e-7a65a92b0c3c/credentials" bf221945-0324-4208-92c7-898726d76692 "https_proxy": "http://aaa.bbb.ccc.ddd:8080" "credhub-ref": "/c/p.spring-cloud-services-scs-service-broker/e9ca123d-c312-4f57-88a4-aef09e887f83/d443c442-c76f-4b7b-b35c-e8227d30958b/credentials-json"
In this example, the app with GUID bf221945-0324-4208-92c7-898726d76692 has both https_proxy and credhub-ref.
It won't be able to interpolate credhub-ref after the upgrade, therefore no_proxy must be configured to contain .cf.internal before the upgrade.
Please restart the app for the change to take effect.