Guide to Troubleshooting TAS Container to Credhub Interpolation Errors

Products

VMware Tanzu Application Service for VMs

Issue/Introduction

Credhub is a component designed for centralized credential management. Credhub is leveraged by many components such as BOSH/Opsmanager and Tanzu Application Service (TAS). For more information on Credhub see this documentation.

In this Knowledge Base article we will be discussing how to troubleshoot TAS application to Credhub communication failures.

To get more insight on how TAS applications leverage TAS Credhub, please see this documentation.

When a TAS container is unable to communicate with Credhub during initial staging and startup it will result in "credhub interpolate" errors. The following error messages may be observed:

[ERR] Unable to interpolate credhub refs: Unable to interpolate credhub references: Post "https://credhub.service.cf.internal:8844/api/v1/interpolate": dial tcp 10.225.57.176:8844: i/o timeout

or

[ERR] Unable to interpolate credhub refs: Unable to interpolate credhub references: Post "https://credhub.service.cf.internal:8844/api/v1/interpolate": dial tcp: lookup credhub.service.cf.internal on 169.254.0.2:53: read udp 10.255.19.105:42496->169.254.0.2:53: i/o timeout

These types of errors are indicative of network connectivity issues. To troubleshoot these types of network connectivity issues, we can push a sample application and perform some network connectivity tests. This is especially helpful for environments using NCP as the overlay networking solution. We will do a walkthrough of this together.

Environment

Product Version: 2.11

Resolution

STEP 1
Push a simple sample application.

Target the org and space you wish to push the application to:
cf target -o "myorg" -s "myspace"

Push the sample app:
mkdir testdir
cd testdir
touch file
cf push sleeper --health-check-type process --buildpack binary_buildpack -c "sleep 10000"

STEP 2
ssh into the app "sleeper" container.

cf ssh sleeper
If cf ssh is not enabled, ssh into the container as root

STEP 3
Test network connectivity to Credhub VMs.

Find the Credhub VM IP addresses:

vcap@f38a6c4d-e1f8-4019-5020-fe5f:~$ nslookup credhub.service.cf.internal
Server:         169.254.0.2
Address:        169.254.0.2#53

Name:   credhub.service.cf.internal
Address: 10.225.57.176

Test network connectivity to each Credhub VM IP address:

Successful example:

vcap@f38a6c4d-e1f8-4019-5020-fe5f:~$ nc -vzw 3 10.225.57.176 8844
Connection to 10.225.57.176 8844 port [tcp/*] succeeded!

Failed Example:

vcap@f38a6c4d-e1f8-4019-5020-fe5f:~$ nc -vzw 3 10.225.57.176 8844
nc: connect to 10.225.57.176 port 8844 (tcp) timed out: Operation now in progress

CONCLUSION
We reviewed a method to test network connectivity between a container and Credhub VMs however it may also be necessary to test the connectivity between the conainer's hosting diego-cell and Credhub VMs. For this, the same instructions can be followed except in step 2 ssh into the diego_cell instead of the application. For example:

STG/0] [ERR] Unable to interpolate credhub refs: Unable to interpolate credhub references: Post "https://credhub.service.cf.internal:8844/api/v1/interpolate": dial tcp 10.225.57.176:8844: i/o timeout
[STG/0] [OUT] Cell 39423f6b-f625-40f1-8ad0-748e47fea470 stopping instance f38a6c4d-e1f8-4019-5020-fe5f

We may want to test network connectivity between diego_cell/39423f6b-f625-40f1-8ad0-748e47fea470 and 10.225.57.176.

There are also other situations where credhub interpolate errors can be observed. See the following KB articles for more information:

CredHub reference can't be interpolated in container due to improper HTTP(S) Proxy Configuration in Tanzu Application Service (TAS) for VMs

App keeps crashing due to error "Unable to interpolate credhub refs ... because the credential does not exist or you do not have sufficient authorization."

Credhub Interpolate times out when pushing an APP in NSX-T environment