Stop unwanted timeout for app health check in TAS for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Most applications must perform an HTTP health check, which tells the platform to send an HTTP request to a specific endpoint of your application. Your application should return an HTTP 200 to this request to signify that it is healthy.

When the health check is not responding fast enough, the application crashes, and the following log message is produced in Diego's BBS log:

Instance became unhealthy: Failed to make HTTP request to '/healthcheck' on port 8080: timed out after 1.00 seconds

Once an application has been deployed on TAS for VMs, Diego performs a health check on the application every second. If the application does not respond within a second to that request, the health check fails.

Environment

Product Version: 2.3

Resolution

Discussed below are three different ways to resolve this issue:

1. Increase the timeout value for the health check on the target application from the default value (1 second) to something larger.

• This will give the application more time to respond.
• The trade-off is that Diego will be a little slower at detecting a problem.

2. Change the application health check endpoint to one that will respond successfully within the default one-second timeout.

• Keep in mind you still want this endpoint to be a good representation of the health of your application.
• The more accurately it represents the health of your application, the more accurately the platform can restart your application when there are problems.

3. Use the port or process-based health check.

• The port health check does not check a specific endpoint, it checks that your application is listening for incoming requests.
• The process health check only checks that the process itself is running.
• The downside of these health checks is that they are not as accurate in measuring the health of your application. There are instances where these health checks pass but your application is not healthy.

To apply these workarounds, you can use the Cloud Foundry (CF) CLI or Apps Manager.

• You can change the health check endpoint through the CF CLI using the set-health-check command.
• You can change the invocation timeout through the CF CLI using the v3-set-health-check command (see the Additional Information section).
• Starting from TAS for VMs v.2.3, you can configure both the invocation timeout and the health check endpoint through Apps Manager, on the Setting tab, for each application.

Additional Information

You can change the invocation timeout via CF CLI with using v3-set-health-check command but it's an experimental feature as of today (December, 2018).