[ERR] Unable to interpolate credhub refs: Unable to interpolate credhub references: Post "https://credhub.service.cf.internal:8844/api/v1/interpolate": dial tcp 10.225.57.176:8844: i/o timeoutor
[ERR] Unable to interpolate credhub refs: Unable to interpolate credhub references: Post "https://credhub.service.cf.internal:8844/api/v1/interpolate": dial tcp: lookup credhub.service.cf.internal on 169.254.0.2:53: read udp 10.255.19.105:42496->169.254.0.2:53: i/o timeout
STEP 1
Push a simple sample application.
Target the org and space you wish to push the application to:
cf target -o "myorg" -s "myspace"
Push the sample app:
mkdir testdir
cd testdir
touch file
cf push sleeper --health-check-type process --buildpack binary_buildpack -c "sleep 10000"
STEP 2
ssh into the app "sleeper" container.
cf ssh sleeper
If cf ssh is not enabled, ssh into the container as root
STEP 3
Test network connectivity to Credhub VMs.
Find the Credhub VM IP addresses:
vcap@f38a6c4d-e1f8-4019-5020-fe5f:~$ nslookup credhub.service.cf.internal Server: 169.254.0.2 Address: 169.254.0.2#53 Name: credhub.service.cf.internal Address: 10.225.57.176
Test network connectivity to each Credhub VM IP address:
Successful example:
vcap@f38a6c4d-e1f8-4019-5020-fe5f:~$ nc -vzw 3 10.225.57.176 8844 Connection to 10.225.57.176 8844 port [tcp/*] succeeded!
Failed Example:
vcap@f38a6c4d-e1f8-4019-5020-fe5f:~$ nc -vzw 3 10.225.57.176 8844 nc: connect to 10.225.57.176 port 8844 (tcp) timed out: Operation now in progress
CONCLUSION
We reviewed a method to test network connectivity between a container and Credhub VMs however it may also be necessary to test the connectivity between the conainer's hosting diego-cell and Credhub VMs. For this, the same instructions can be followed except in step 2 ssh into the diego_cell instead of the application. For example:
STG/0] [ERR] Unable to interpolate credhub refs: Unable to interpolate credhub references: Post "https://credhub.service.cf.internal:8844/api/v1/interpolate": dial tcp 10.225.57.176:8844: i/o timeout [STG/0] [OUT] Cell 39423f6b-f625-40f1-8ad0-748e47fea470 stopping instance f38a6c4d-e1f8-4019-5020-fe5f
We may want to test network connectivity between diego_cell/39423f6b-f625-40f1-8ad0-748e47fea470 and 10.225.57.176.
There are also other situations where credhub interpolate errors can be observed. See the following KB articles for more information:
CredHub reference can't be interpolated in container due to improper HTTP(S) Proxy Configuration in Tanzu Application Service (TAS) for VMs
App keeps crashing due to error "Unable to interpolate credhub refs ... because the credential does not exist or you do not have sufficient authorization."
Credhub Interpolate times out when pushing an APP in NSX-T environment