GPSS unexpected trigger of fallback_offset resets offset to minimum value in RabbitMQ

Products

• VMware Tanzu Data Intelligence
• VMware Tanzu Greenplum
• VMware Tanzu Greenplum / Gemfire
• VMware Tanzu Data Suite
• VMware Tanzu Data Suite

Issue/Introduction

When using Greenplum Streaming Server (GPSS) to consume data from a RabbitMQ cluster, the job experiences an unexpected trigger of the fallback_offset configuration. This behavior results in the processed stream offset resetting to the minimum value stored in RabbitMQ, which causes duplicate data to be loaded into the database. This issue typically manifests during a job restart or when connection instability occurs, occasionally generating the following connection timeout error:

dial tcp <load_balancer_ip>:<stream_port>: connect: connection timed out

Environment

GPSS 1.X

Cause

This issue is caused by a hardcoded timeout mechanism combined with the fallback logic in GPSS.

When a GPSS job resumes, it successfully reads the last committed end offset from the history table. To verify this offset is still valid and has not been truncated by RabbitMQ's retention policy, GPSS queries the current offset range from the broker by spawning a temporary consumer. This query has a hardcoded 500-millisecond timeout.

If there is minor network jitter, high broker load, or a large message chunk being transmitted, the temporary consumer fails to fetch a message within 500ms. Consequently, the function returns -1 for the earliest offset. GPSS incorrectly concludes that an "Offset Gap" exists. Because the job configuration contains FALLBACK_OFFSET: earliest, GPSS discards the valid history offset and silently resets the cursor to the very beginning of the stream, leading to duplicate data loading.

Resolution

Workaround: To prevent this issue in the current version, modify the job configuration to remove the fallback offset parameter:

1. Open the GPSS job configuration file.

2. Remove the line FALLBACK_OFFSET: earliest or set it to an empty value ("").

3. Save the configuration.

4. Stop the currently running job if it is loading duplicate data.

5. Restart the job.

Permanent fix is targeted to be fixed in a future release GPSS 2.3.3