High CPU and memory usage on Greenplum Streaming Server (GPSS) hosts.

Products

• VMware Tanzu Greenplum

Issue/Introduction

The amount of memory and CPU used by the GPSS service can grow very large over time.
The output from "top" command would show high virtual memory (VIRT) usage.

%Cpu(s): 5.0 us, 27.7 sy, 0.0 ni, 67.0 id, 0.1 wa, 0.0 hi, 0.2 si, 0.0 st
KiB Mem : 32761604 total, 17338576 free, 2199696 used, 13223332 buff/cache
KiB Swap: 8389628 total, 5601556 free, 2788072 used. 29956160 avail Mem

PID    USER     PR  NI  VIRT   RES     SHR  S  %CPU   %MEM  TIME+     COMMAND
44720  gpadmin  20  0   69.2g  637488  6492 S  132.7  1.9   13650:12  gpss

The virtual memory is amount of RAM and swap used by the process.
The high memory usage can be seen even when there are no jobs running.
 

This can be seen when reading data from Kafka. GPSS server may not free some memory when there is offset gap error or it fails to get partition metadata.
There are some methods to avoid this issue before new version released:

1. avoid offset gap on starting a job

2. make sure all the brokers available by the configured connection method(SSL/SASL/..)

Environment

Product Version: 6.22

Resolution

Fix for the issue will be released in GPSS 1.8.1 and 1.9.0 and above