How to collect High Memory Usage in Greenplum gpmon/gpperfmon Processes Using MemoryContextStats
Article ID: 406725
Updated On:
Products
Issue/Introduction
Purpose: This article guides you through diagnosing high memory usage issues observed in Greenplum Database monitoring processes such as gpmon and gpperfmon. It involves identifying problematic processes, capturing memory context statistics, and collecting relevant logs and artifacts to support troubleshooting.
TopMemoryContext in Greenplum serves as the root memory context in the hierarchical memory management system inherited from PostgreSQL. It acts as the highest-level memory context that lasts for the entire lifetime of a backend process. This context is the parent from which all other child memory contexts are created and managed throughout the query execution and process lifecycle.
The role of TopMemoryContext in Greenplum's memory management system includes:
-
Root of Memory Context Tree: It is the top-most context that remains active for the whole backend process and holds memory allocations that need to persist for that duration.
-
Memory Allocation Grouping: It groups all memory allocations into a hierarchy, allowing for efficient bulk freeing of memory by releasing entire contexts and their children at once.
-
Memory Leak Avoidance: By using this hierarchical memory context system, Greenplum prevents memory leaks and simplifies memory cleanup, especially in error handling or after query execution.
-
Managing Segment Memory: It helps Greenplum segments manage memory safely and efficiently during query processing, enabling better control of memory usage at a process level.
Thus, TopMemoryContext is foundational for how Greenplum manages memory internally, making query execution more efficient and stable by controlling the lifecycle and size of memory allocations.
Environment
GPDB: 6.x.x
Greenplum Command Center
Resolution
Step-by-Step Procedure
1. Identify gpmon/gpperfmon Processes with High Memory Usage. Run the following command on the Greenplum segment host(s) to list processes and identify those with high memory usage:
Look for processes named gpmon or gpperfmon with elevated RES (resident memory) or %MEM values.
Note: the corresponding Process IDs (PIDs) of these high-memory processes.
ps aux
2. Attach to the Process Using gdb. For each PID identified, attach the GNU Debugger (gdb) to the process:
This enables interaction with the live process for further inspection.
gdb -p <PID>
3. Obtain Memory Context Statistics. Within the gdb interactive session, run the following command:
This command prints detailed statistics about memory contexts rooted at TopMemoryContext. The output is logged into the corresponding Greenplum segment’s backend log file.
p MemoryContextStats(TopMemoryContext)
Exit gdb after the command completes.
4. Collect Greenplum Logs. Gather the following logs for analysis:
- Segment logs that contain the MemoryContextStats output.
- The Master logs covering the same time range as when the memory issue was observed.
- If a core dump for any of these processes is available, please collect and provide it as this can tremendously assist in root cause analysis.
5. Provide GPCC Artifacts. Greenplum Command Center (GPCC) artifacts are helpful for correlating monitoring data.
On the host where GPCC is running, run:
ls -altr /bb/bin/gplum/greenplum-cc-6.x.x/ccdata/
Send the output of this command along with any relevant files.
Specifically, please share all files starting with:
gpcc_queryinfo_*
If these files are very large, provide at least one example of each file type:
gpcc_queryinfo_*.dat
gpcc_queryinfo_*.dat.ext
6. Provide GPCC Logs. Collect GPCC logs including but not limited to:
webserver.log
Ensure logs cover the timeframe when the issues started onward to capture any errors or warnings related to the issue.
Feedback
