"Exit status 137" message seen in Tanzu Application Service for VMs logs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

The commands, cf events and cf logs, show entries with the message "Exit status 137" in Tanzu Application Service for VMs (TAS for VMs).

This article explains some of the situations where this exit status is seen and the logging pattern associated with each situation.

Environment

TAS/EAR

Resolution

The exit code 137 corresponds to 128 + SIGKILL (9).

Container memory limit

When the container memory limit is hit:

• cf events: Exited with status 137 (out of memory)
• cf logs: Exit status 137 (out of memory)

Demo

$ mktemp -d /tmp/XXXXX
$ touch /tmp/S1k0S/file
$ cf push ContainerLimit -p /tmp/S1k0S -m 8M -b binary_buildpack -u process -c "sleep 30 | </dev/zero head -c 50M | tail"
$ # Wait 30 seconds
$ cf logs ContainerLimit --recent | grep APP/PROC/WEB | grep -F "Exit status"
   2021-09-13T15:32:16.09+0000 [APP/PROC/WEB/0] OUT Exit status 137 (out of memory)
$ cf events ContainerLimit | grep -F "Exited with status"
2021-09-13T15:32:21.00+0000   audit.app.process.crash     web               index: 0, reason: CRASHED, cell_id: 7e660bde-728d-4276-8ebb-c8f9466c7fc9, instance: 4ec527c0-756f-4a0c-6f1d-1e0b, exit_description: APP/PROC/WEB: Exited with status 137 (out of memory)
2021-09-13T15:32:21.00+0000   app.crash                   ContainerLimit   index: 0, reason: CRASHED, cell_id: 7e660bde-728d-4276-8ebb-c8f9466c7fc9, instance: 4ec527c0-756f-4a0c-6f1d-1e0b, exit_description: APP/PROC/WEB: Exited with status 137 (out of memory)

JVM memory limit 

When the JVM memory limit is hit:

• cf events: Exited with status 137 (without "(out of memory)")
• cf logs: Exit status 137 (without any "(out of memory)", alongside "jvmkill killing current process")

Demo

$ cd ~/TAS_java-app/
$ mvn package
$ cf push JVMLimit --manifest manifest.yml
$ curl -k  https://JVMLimit.cfapps-02.slot-59.pez.vmware.com/test
$ cf events JVMLimit | grep -F "Exited with status"
2021-09-13T16:15:01.00+0000   audit.app.process.crash     web        index: 0, reason: CRASHED, cell_id: 7e660bde-728d-4276-8ebb-c8f9466c7fc9, instance: 8f758165-d8a7-4168-4a44-66de, exit_description: APP/PROC/WEB: Exited with status 137
2021-09-13T16:15:01.00+0000   app.crash                   JVMLimit   index: 0, reason: CRASHED, cell_id: 7e660bde-728d-4276-8ebb-c8f9466c7fc9, instance: 8f758165-d8a7-4168-4a44-66de, exit_description: APP/PROC/WEB: Exited with status 137
$ cf logs JVMLimit --recent | grep APP/PROC/WEB | grep -E "Exit status|jvmkill kill"
   2021-09-13T16:14:50.74+0000 [APP/PROC/WEB/0] ERR jvmkill killing current process
   2021-09-13T16:14:55.74+0000 [APP/PROC/WEB/0] ERR jvmkill killing current process
   2021-09-13T16:14:55.75+0000 [APP/PROC/WEB/0] OUT Exit status 137

Note: TAS_java-app.zip is attached to this KB to allow replicating the below steps. Please note that this test app is not a VMware product, is only used here as an example and does not come with any guarantee.


Shutdown timeout

When the shutdown timeout is hit:

• cf events: Nothing in case of a cell drain, the scaling event in case of scale/autoscale down of an app.
• cf logs: Exit status 137 (exceeded 10s graceful shutdown interval)

When a container is stopped, if the graceful shutdown (with SIGTERM) doesn't complete within the configured timeout (10 seconds by default), it is killed. For more information, refer to Shutdown.

This may happen during operations such as the drain of a cell, or a scale/autoscale down of an app.

Note: If the graceful shutdown was completed successfully, the exit status would be 143 = 128 + SIGTERM (15).

Demo (Cell Drain)

$ mktemp -d /tmp/XXXXX
$ touch /tmp/QWh2V/file
$ cf push UngracefulShutdown -p /tmp/QWh2V -m 8M -b binary_buildpack -u process -c "trap -- '' SIGTERM; while true; do sleep 1s; done"
$ # Find the cell running the app instance
$ cf logs UngracefulShutdown --recent | grep Cell | tail -n 1
   2021-09-14T12:29:41.33+0000 [CELL/0] OUT Cell c35047b2-1dc9-4774-8cc3-05fe18e734af successfully created container for instance 06f1ce25-0019-4158-713d-6136
$ # Drain that cell
$ bosh -d cf-948e165e6fa390e9cd35 ssh compute/c35047b2-1dc9-4774-8cc3-05fe18e734af -c "sudo /var/vcap/jobs/rep/bin/drain"
$ # cf logs show "Exit status 137 (exceeded 10s graceful shutdown interval)" on the app
$ cf logs UngracefulShutdown --recent | grep APP/PROC/WEB | grep -F "Exit status"
   2021-09-14T12:30:51.45+0000 [APP/PROC/WEB/0] OUT Exit status 137 (exceeded 10s graceful shutdown interval)
$ # cf events show nothing
$ cf events UngracefulShutdown | grep -F "Exited with status"
$ cf events UngracefulShutdown | grep -F "2021-09-14T12:30"
$