GPSS 2.3.2: Kafka Consumer Offsets Reset to Earliest Upon Job Restart when AUTO_SCALE is Enabled

Products

• VMware Tanzu Data
• VMware Tanzu Greenplum
• VMware Tanzu Greenplum / Gemfire

Issue/Introduction

When running a Greenplum Streaming Server (GPSS) 2.3.2 job with the AUTO_SCALE feature enabled, restarting the stream causes the Kafka consumer offsets to reset to FALLBACK_OFFSET, which could be set to "earliest" or "latest"

Instead of resuming ingestion from the last committed offset, the job initiates from the FALLBACK_OFFSET of the topic partitions, which can be set to . This results in significant consumer lag and potential duplication of data processing.

Environment

• Product: Greenplum Streaming Server (GPSS)
• Version: 2.3.2
• Deployment: Kubernetes
• Configuration: AUTO_SCALE: ENABLED: true

Cause

This behavior is caused by a confirmed product defect in GPSS 2.3.2, where the internal job logic incorrectly handles Kafka offset persistence during restarts specifically when the auto-scaling mechanism is active.

Resolution

Workaround

To prevent the offset reset and maintain stream continuity, disable the AUTO_SCALE feature in your GPSS job configuration.

Modify your YAML file to disable AUTO_SCALE:

 

AUTO_SCALE: 
   ENABLED: false

Once this parameter is updated, the GPSS job will correctly track and preserve consumer offsets across job restarts and state transitions.

Fix

This issue is a tracked product defect. A code fix will be included in a future maintenance release of the Greenplum Streaming Server. Please monitor the release notes for subsequent versions for details on this fix.