App container fails to start due to error "Unable to interpolate credhub references ... connect: connection refused"

Products

• VMware Tanzu Application Service

Issue/Introduction

Application container initialization fails with following error message in app log. 

2025-04-12T11:04:32.145+01:00 [APP/PROC/WEB/0] [OUT] Failed on attempt 1 out of 3: Unable to interpolate credhub references: Post "https://credhub.service.cf.internal:8844/api/v1/interpolate": dial tcp ###.###.###.###:8844: connect: connection refused

Cause

If an application is binding with backend services, and credhub integration is enabled with the services, during the application container initialization, a process will interpolate service binding credentials with Post request "https://credhub.service.cf.internal:8844/api/v1/interpolate". "dial tcp ###.###.###.###:8844: connect: connection refused" indicates the failure is due to network layer problem. Possible causes are: 

• Crehub job is down, thus no process is listening at port 8844.
• The connection is refused by firewall rules.

Resolution

In order to identify the root cause and resolve it, please follow steps:

1. Check if credhub job is running at credhub VM with `monit summary`, review credhub log and resolve any problem if the job is failing.
2. Check if credhub process is listening at port 8844 with `netstat -lnpt`, review credhub log and resolve any problem if the job is failing.
3. Check the network connection from diego_cell to credhub port 8844, check both networking routing with `traceroute` and TCP connection `nc -zv <CREDHUB_IP> 8844`.
4. If there is no problem from diego_cell to credhub port 8844, the problem is most likely with application security groups(ASGs), please check with `cf security-groups`, `cf security-group NAME`. Usually it is due to traffic from containers to destination IPs (credhub VM IPs) are not allowed by ASGs. Modifying ASGs should be able to resolve the connection error.