Application instances CRASHED. 'APP/PROC/WEB: Exited with status 137 / 134 after rolling back java buildpack version

Products

• VMware Tanzu Platform - Cloud Foundry
• VMware Tanzu Application Service
• VMware Tanzu Application Catalog

Issue/Introduction

You rolled back an applications Java buildpack version and restarted the app.  The app starts crashing even though application instances have 8gb memory each.  When checking the application logs, you find the below errors/exit status:

APP/PROC/WEB: Exited with status 137
APP/PROC/WEB: Exited with status 134

Cause

When an application throws both of these codes simultaneously, then understanding and addressing both individually is key. Start with status 137, then move to addressing 134.

1.) Exited with status 137 (sigkill) -  Java 8 is less efficient at handling memory and requires more than it's now allocated, so runs out of memory and throws a status 137. Exploring further in the app logs should reveal which resource/setting is hitting/exceeding memory limits (as seen in the snippet below):

Hint: Metaspace is over 95% full. To increase it, set -XX:MaxMetaspaceSize to a suitable value.

 

2.) Exited with status 134 (sigabrt) - The app was compiled on java 21 but is running on Java 8 runtime, which can trigger the Java Virtual Machine (JVM) to hit issues such as class incompatibilities, Bytecode, etc.   This causes the JVM's 'Just In Time' compiler (JIT) to fail with a safety assertion error.  The JVM will throw the below error if this issue is encountered:

# snippet from the JVM logs:
'(!thread->is_Compiler_thread()) failed: cannot make java calls from the compiler'

Resolution

To resolve, address exit status 137, then 134.

Follow the below steps in order:

1.) Exited with 137: 

• Configure the applications User Provided Environment Variables per container memory & Java version compatibility requirements. Specifically the following java properties -Xms, -XX:MaxMetaspaceSize, and -XXMaxDirectMemory Size. These values are calculated and determined by the user based on the applications requirements per said java property(s). These properties can be configured by using cf command(s) and/or utilizing AppsManger UI (see Examples below):
  • A.) cf command:

cf set-env <app-name> JAVA_OPTS "-XX:MaxMetaspaceSize=256m"

• • B.) appman UI:

 

 

 

 

 

2.) Exited with 134: 

• Restage the app, not restart. Restarting the app leaves stale compiled code (from java 21) in the app which is incompatible with the targeted version (java v8). Restaging the app will form a new droplet, pick up the correct java memory properties from the newly/properly configured User Environment Variables performed in step 1 above.

 

Additional Information

See the official Java Properties Information page for detailed information.