Application crashes with "Resource exhaustion event: the JVM was unable to allocate memory from the heap." error in Tanzu Application Service for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Your application (app) crashes with the error:

Resource exhaustion event: the JVM was unable to allocate memory from the heap.

When you check your app logs to get more info with the command, cf logs <APP-NAME> --recent, you notice this error:

2021-07-19T16:26:10.343+05:30 [APP/PROC/WEB/0] [ERR] Resource exhaustion event: the JVM was unable to allocate memory from the heap.
2021-07-19T16:26:10.343+05:30 [APP/PROC/WEB/0] [ERR] ResourceExhausted! (1/0)

Your app is probably creating a lot of objects which increase your applications memory usage. This might be due to increase of traffic or this might also signify a memory leak. To address resource exhaustion you need to understand your applications memory usage.

Environment

Product Version: 2.10

Resolution

The easiest way to resolve this is to allocate more memory to your app with the below cf scale command.  The JAVA Memory Calculator will automatically scale up memory settings using the larger pool of memory: 

cf scale -m 2G

Note: The increase of memory might be normal, specially if you have more requests or traffic and your app tends to increase objects that are created.

If scaling the memory does not help there could be other memory settings you need to adjust or a possible memory leak.

Memory Leak Scenario 

If scaling memory does not help and you suspect that there might be a memory leak, keep the following in mind when troubleshooting your application.

• Monitor your application with APM tools, such as NewRelic,AppDynamic or Jconsole, and observe your application's memory behavior.
• Check you application logs. When a resource exhaustion is thrown it also prints a histogram. Check the histogram for large objects in terms of both in total bytes and instance count.

2021-07-19T16:26:12.580+05:30 [APP/PROC/WEB/0] [OUT] | Instance Count | Total Bytes | Class Name |
2021-07-19T16:26:12.581+05:30 [APP/PROC/WEB/0] [OUT] | 183478 | 907825328 | [B |
2021-07-19T16:26:12.582+05:30 [APP/PROC/WEB/0] [OUT] | 262144 | 6291456 | Lorg/apache/logging/log4j/core/async/AsyncLoggerConfigDisruptor$Log4jEventWrapper; |
2021-07-19T16:26:12.583+05:30 [APP/PROC/WEB/0] [OUT] | 172378 | 4137072 | Ljava/lang/String; |
2021-07-19T16:26:12.585+05:30 [APP/PROC/WEB/0] [OUT] | 126706 | 4054592 | Ljava/util/concurrent/ConcurrentHashMap$Node; |
2021-07-19T16:26:12.586+05:30 [APP/PROC/WEB/0] [OUT] | 27712 | 3312864 | Ljava/lang/Class; |
2021-07-19T16:26:12.587+05:30 [APP/PROC/WEB/0] [OUT] | 8242 | 3061496 | [I |
2021-07-19T16:26:12.589+05:30 [APP/PROC/WEB/0] [OUT] | 13487 | 2173808 | [Ljava/lang/Object; |
2021-07-19T16:26:12.590+05:30 [APP/PROC/WEB/0] [OUT] | 50043 | 1601376 | Ljava/util/HashMap$Node; |
2021-07-19T16:26:12.591+05:30 [APP/PROC/WEB/0] [OUT] | 16914 | 1488432 | Ljava/lang/reflect/Method; |
2021-07-19T16:26:12.592+05:30 [APP/PROC/WEB/0] [OUT] | 16387 | 1436328 | [Ljava/util/HashMap$Node; |
2021-07-19T16:26:12.594+05:30 [APP/PROC/WEB/0] [OUT] | 1110 | 1243176 | [Ljava/util/concurrent/ConcurrentHashMap$Node; |
2021-07-19T16:26:12.595+05:30 [APP/PROC/WEB/0] [OUT] | 29619 | 1184760 | Ljava/util/LinkedHashMap$Entry; |
2021-07-19T16:26:12.596+05:30 [APP/PROC/WEB/0] [OUT] | 30764 | 984448 | Ljava/lang/ref/WeakReference; |
2021-07-19T16:26:12.597+05:30 [APP/PROC/WEB/0] [OUT] | 783 | 924688 | [C |

 

Note: In the sample above,  [C and [L and [I. represents the encoding of the element type. For more information, refer to getName and element table.

For more information on the histogram, refer to 2.14.2 Heap Histogram.

• Take various heap dumps in your app as the memory usage increases. For example, take 2 or 3 heap dumps with 30-60s interval. For more information on taking various heap dumps, refer to How to generate and download Java Application heap dump from a Tanzu Application Service for VMs container. You can run it to an analyser, such as yourKit, and compare heap dumps to check which objects increases over time this will give you an idea where in your code might have memory leak.

 

MaxMetaspaceSize is too low

In some cases the MaxMetaspaceSize is too low for applications with large dependencies. Developers can override the JAVA Memory Calculator settings using the java opts environment variable.

cf set-env my-app JAVA_OPTS '-XX:MaxMetaspaceSize=256m'