How to troubleshoot an application that fails only when run on VMware Tanzu Application Service for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

For a number of reasons, it is possible to have an application that runs fine locally but does not run on VMware Tanzu Application Service for VMs (TAS for VMs). This article will walk through the differences between your application running locally and on TAS for VMs, and it will show how to troubleshoot the application.

Resolution

As previously mentioned, an application running locally on a developer's workstation is not the same as an application running on TAS for VMs. Even if you have the same bits of code, there are differences which can cause problems. The following list summarizes many of the common differences you might encounter.

Language / Runtime Differences

When running an application locally, a specific version of the application's language or the runtime is installed. The version may or may not be the one being used on TAS for VMs and can possibly have incompatibilities. If there are application failures on the TAS for VMs, compare the version of the language or the runtime being used locally against the version on TAS for VMs.

The process for determining the version varies by runtime. Some common strategies for locating the runtime version on TAS for VMs include looking through the buildpack's output while running cf push (which often shows the version) or grabbing the version programmatically and then, writing it to a log file or a web endpoint.

In addition to the language or the runtime version, pay attention to the version of any server software being installed by the buildpack to deploy the application. This is most relevant to Java users who have their application deployed in Apache Tomcat, and PHP users who have their application deployed behind Nginx or Apache HTTPD. Make sure that the local environment matches the TAS for VMs environment as closely as possible to minimize the possibility of problems.

Memory Limit

When you deploy an application to TAS for VMs, you will select a memory limit for the application. The memory limit sets an upper bound on how much memory the application can access when it runs. If the application attempts to exceed this limit by any amount, the platform will kill the application and restart it.

As the system simply kill's the process when the application exceeds the memory limit, it can sometimes be confusing when an application crash. Both result in the application going down and being restarted, however, running out of memory will result in an exit code of 137 (exit codes are visible by running cf events or cf logs) and the message (out of memory) whereas an application crash would have some application specific, non-zero exit code.

If there is an unexplained crash when deploying the application to TAS for VMs or if you see an exit code of 137 and the message (out of memory), then increase the memory limit for the application. Once the application stabilizes use cf app to view memory usage or enable monitoring with a APM tool to further investigate the memory needs of your application.

Not Listening for a Connection

With many languages (Go, Python, Ruby, Node.js, etc.), it is the responsibility of the application to listen for incoming HTTP requests (i.e. there is no application server like with Java or PHP). For languages like this, it is perfectly acceptable for an application running locally to listen on a common port like 5000, 8000 or 8080, but, this is not acceptable when running on TAS for VMs.

The TAS for VMs platform does not allow the individual applications to pick the port where it listens. Instead, the platform assigns a port to the application (through the PORT environment variable) and requires the application to listen to that specific port. If your app does not listen to the assigned port, the system thinks it did not start or has crashed.

If you have an application that directly listens for incoming HTTP requests, make sure that it is listening to the correct port by examining the PORT environment variable and listening to the port contained in it.

Health Checks

When your application runs on TAS for VMs, you will define a health check for it. The health check is used by the platform to determine that your application is healthy. The app must pass its initial check within 60 seconds (up to 180 seconds, controlled with cf push -t <seconds>) or the platform will consider it to have crashed and will restart it. Plus, the app must continue to pass its health check every 30 seconds going forward for the rest of its lifetime. If the application ever fails a health check, the platform will consider it to have crashed and will restart it.

For more information on health checks, such as how to set them or what types are available please see the documentation here.

If your application is failing its initial health check, you will see an error like Instance never healthy after 1m0s: Failed to make TCP connection to port 8080: connection refused.

If your application fails its health check after it has been running, you will see an error like Instance became unhealthy: Failed to make HTTP request to '/healthcheck' on port 8080: timed out after 1.00 seconds.

You may also run cf events app-name to see if your application is crashing due to failed health checks.

Services

If your application consumes services, there will almost certainly be differences between developing locally and running your application on TAS for VMs. You may have different service implementations (MariaDB vs MySQL) or different service versions. You might also have different levels of access to the service (total control locally vs restricted access in production). These can uncover subtle differences in behavior and sometimes cause errors. If your application does not start, please check for errors talking to the service.

In addition, how you obtain the service credentials is likely different locally versus running on TAS for VMs. When running locally, you are probably obtaining credentials from a local config file or based on framework defaults (like localhost:3306). When running on TAS for VMs, you are almost certainly using a service that is bound to the application and reading credentials from VCAP_SERVICES. If your application does not start, please validate that credentials are being loaded correctly. An easy way to do this, is to temporarily add an additional logging statement and log them out after read by the application (remove this when done, as you should not log credentials).

Lastly, your services are probably running locally when you are developing your application. This means there is effectively no network in play. When you deploy to TAS for VMs, your services may run on different networks. This means you can have connection problems and latency. If your application does not start, please validate that you are able to make a connection to your service from TAS for VMs. An easy way to do this is by using cf ssh to enter a running container in your org and space (it doesn't have to be the application itself, any container in the same org and space is fine). Then run the command nc -v <service-ip> <service-port>. This will make a TCP connection to your service. If it is successful and as fast as you would expect then you can rule out networking. If it is slow, errors or hangs, then you have network problems. Please review your foundation's application security groups and also the network path between your application and the service.

Operating System

When your application runs locally, it will run on whatever operating system you are using. That might be Windows, MacOS or any number of Linux distributions. When your application runs on TAS for VMs it will run in a container that is utilizing the cflinuxfs3 stack, which is built from Ubuntu 18.04. The differences are obvious when you compare Windows to the cflinuxfs3 stack, but there can be subtle differences between cflinuxfs3 and MacOS or even different distributions of Linux.

Applications which are compiling native code or that depend on specific shared libraries can be particularly fragile. You will need to take caution that such code is compiled and tested against Ubuntu 18.04.

Conditional Configuration

Many frameworks have provisions to conditionally enable features based on the environment (i.e. dev, test or prod). For example, the Spring Framework has profiles which  can be used to conditionally configure parts of your application. While this functionality is convenient and helpful to application developers, it can make troubleshooting issues more difficult because your application can function drastically different based on these the active environments and profiles.

To rule out conditional logic as a possible failure cause, please make sure that you are using the same environments and profiles when you run your application locally that will be used when run on TAS for VMs. In addition, for Spring users, please be aware that the Java buildpack will by default enable the "cloud" profile. When debugging a problem with your application locally, please make sure you enable this profile so that you get a consistent experience.

Troubleshooting Failures

The recommended approach for troubleshooting the situation where an app runs locally but not on TAS for VMs is to either review the items above and isolate the ones that impact your app, which can be tedious, or to remove the differences. The latter can be achieved by using Docker and the cf cli plugin called cflocal.

The cflocal plugin will deploy your application in a way that is very similar to how the application is deployed on TAS for VMs. For example, it will use the same cflinuxfs3 stack, the same buildpacks, the same memory limits and it can even be configured to pass in service information using the same method as TAS for VMs. This removes many of the differences discussed above, which allows you to narrow down on what is different and causing problems with your application.

Instructions

1. Install Docker CE on your workstation.

2. Install the cf cli, if it's not already there.

3. Install the cflocal plugin with the command cf install-plugin -r CF-Community "cflocal".

4. Create a local.yml file for your application. This looks very similar to an application's manifest.yml. Here is an example:

---
applications:
- name: my-super-cool-app
  buildpacks:
  - some_buildpack
  memory: 2G
  disk_quota: 4G
  staging_env:
    SOME_STAGING_VAR: "some staging value"
  running_env:
    SOME_RUNNING_VAR: "some running value"
  env:
    SOME_VAR: "some value"

5. Then run cf local stage my-super-cool-app. This will run the buildpacks and stage your app in a local container. If there are issues, this can be easier to debug. Furthermore, you end up with a file my-super-cool-app.droplet in the current directory that you can inspect.

6. You can run cf local run my-super-cool-app to run the application. You can then access it on localhost and the port listed in the output.

a. In addition, you can set the -t flag to start a shell within the application container to debug.
b. You can also set -s app to use service bindings from a remote app deployed to

TAS for VMs and you can use -f app to tunnel service connections through an app. Both of these can be used to minimize the difference between consumed services. 7. Two additional options that can be helpful for troubleshooting:

a. You can use cf local push my-super-cool-app to deploy the droplet directly to a remote TAS for VMs instance.

b. You can use cf local export my-super-cool-app to generate a Docker image from the droplet. You can then use docker run to execute directly in Docker.