User realize they were using -Dgemfire.disableConflictChecksOnClient=true in GemFire client JVM, needed to understand what it does and why it was being used.
GemFire 9.x
GemFire 10.x
disableConflictChecksOnClient flag is used for whether client should detect there is a commit conflict already based on other transactional operations on the same client. The default setting is false, which it throws a fail fast exception (without going through the normal checking through DLock on servers).
If the flag is turned on, it will always go to the servers and check the conflict using DLock services).
The only scenario I can think of that has some benefits of turning this flag on is the following:
Txn1 on client1 updates on k1, and k2 (holding the local locks)
txn2 on client1 updates on k2 (if disable the conflict checks on client is on), the txn2 will be forwarded to server (without fail fast).
Meanwhile, Txn1 failed on conflict check (assuming another transaction by another client already holding the lock on k1).
In this case, Txn1 on client1 will fail, but Txn2 on client1 will succeed (assuming no other transactions holding the lock on k2)