GemFire: Data Inconsistency Due to Transaction Expiration After Client Disconnects

Products

• VMware Tanzu Data Intelligence
• VMware Tanzu Data Suite
• VMware Tanzu Data Suite
• VMware Tanzu Gemfire
• VMware Tanzu Greenplum / Gemfire

Issue/Introduction

This issue can occur when a client-initiated transaction runs for an extended period of time, more than 3 minutes, and the client becomes temporarily disconnected from the GemFire server during transaction execution. In such cases, earlier operations in the transaction may be lost even though the transaction eventually commits.

Cause

GemFire is designed with the assumption that transactions are short-lived. To prevent unfinished or abandoned transactions from consuming memory indefinitely, GemFire automatically expires transactions when client activity stops.

Behavior:

• When a transaction is initiated by a client, GemFire tracks client activity.

• If the client becomes disconnected and does not perform the next transactional operation within 3 minutes, GemFire assumes the client is no longer active.

• After this timeout, GemFire expires the transaction and cleans up its internal state.

• The timeout is controlled by the system property:

  gemfire.cacheServer.transactionTimeToLive

  • Default value: 180 seconds (3 minutes)

If the client reconnects after the transaction has already expired and continues issuing operations:

• GemFire treats the activity as a new transaction

• Only the operations performed after the reconnect are committed

• Operations performed before the disconnect are lost, resulting in partial transaction commit

This behavior is most commonly seen when transactions run significantly longer than the default timeout (for example, 10–15 minutes).

Resolution

Workaround

For environments that require long-running client transactions, increase the transaction timeout to exceed the maximum expected transaction duration:

 

 gemfire.cacheServer.transactionTimeToLive 

Please note: Increasing this value may increase memory usage if a client is permanently disconnected without committing or rolling back a transaction. Transaction memory is released only when:

• The client commits or rolls back, or

• GemFire expires the transaction

Product Fix (Upcoming Release)

This fix is tracked under GEM-16988 and is planned for the following releases:

• GemFire 10.3.0

• GemFire 10.2.2

• GemFire 10.1.7