How to identify out of memory (OOM) errors

Products

VMware Tanzu Greenplum VMware Tanzu Data Suite VMware Tanzu Data Suite Greenplum

Issue/Introduction

There are several reasons a Greenplum Database (GPDB) process generates an out of memory (OOM) error. GPDB attempts to provide meaningful error codes and messages that detail why an action resulted in an OOM.

These errors fall into two categories and are identified by the associated error codes. Within the two categories, there are a number of reasons for the generated error.

The following are examples of possible OOM errors. Note that the error codes are 53400 and 53200.

"ERROR","53400","Out of memory (seg13 slice13 sdw1:40001 pid=10183)","VM Protect failed to allocate 8388608 bytes, 6 MB available"

"ERROR","53200","Out of memory. Failed on request of size 156 bytes. (context 'CacheMemoryContext') (aset.c:840)"

Memory Error Code 53400

This error is produced when there is an issue allocating memory under the gp_vmem_protect_limit while the query is being terminate.

"VM Protect failed to allocate %d bytes,%d MB available": This happens when the request results in exceeding the gp_vmem_protect_limit.
"Per-query VM protect limit reached: current limit is %d kB, requested %d bytes, available %d MB": The following limit is exceeded: gp_vmem_protect_limit_per_query.
"VM protect failed to allocate %d bytes from system, VM Protect %d MB available": This is not a result of reaching the gp_vmem_protect_limit. Rather, the error is due to the OS not having enough user memory available.
"Failed to allocate memory under virtual memory protection" "Error %d, errno %d, %s": This error is due to a memory allocation problem that is not covered by the three cases above.

Environment

Resolution

Solutions to error code 53400

The most common cause for this error is due to:

a. Resource queues are misconfigured, leading to concurrency being too high or memory oversubscribed.

b. A query plan is too large and is consuming too much memory.

For the first case please review the resource queue settings.

For the large query plan size issue, consider implementing the GUC gp_max_plan_size.

Note: Reaching gp_vmem_limit_per_query value is due to overly large query plans. gp_vmem_limit_per_query is only available in GPDB 5.x

The reason an OS is reporting an OOM is one of the following:

a. A misconfigured server is oversubscribe to memory (the gp_vmem_protect_limit is too high for the number of segments).

b. Another process is consuming the system memory. Consult with Pivotal Support if there is trouble finding the cause.

Note: If another process is consuming the system memory, please open a support ticket.

Error Code 53200

This error covers a large number of possible error messages. They all relate to the postmaster process not being able to allocate memory directly from the OS. The exact cause will depend on the situation and the error message provided.

Additional Information

If you need further assistance, contact Broadcom Support.