Perform Cassandra Query assertion not respecting change in query consistency

Products

• CA API Gateway

Issue/Introduction

We have recently tested the shutdown of one half of our cassandra cluster and found that the query consistency setting is not getting updated.

We have:

DSE6.8.4

Three cassandra nodes in STL

Three cassandra nodes in KC

The Cassandra connection properties for all kepsaces we use are pointed to two of the three KC nodes.

When performing a query assertion while all nodes running we are successfull

After shutting down the STL datacenter nodes we have failure in authentication

Perform Cassandra Query" assertion failed due to: Not enough replicas available for query at consistency SERIAL (4 required but only 3 alive)

This is expected. So we changed the cassandra query consistency level by unchecking the default box and setting the query conistency drop down to local_quorum. And of course saving and activating the policy. 

We then shut down the STL datacenter nodes, and see that the assertion continues to fail with the same audit:

Perform Cassandra Query" assertion failed due to: Not enough replicas available for query at consistency SERIAL (4 required but only 3 alive)

The cluster wide property for consistency is set to ONE which is the default and has not been touched. This appears to be a bug in the the product. We see that the revision history of the policy is updated so this looks like a failed implementation of the assertion to switch the consistency setting. 

Environment

Release : 10.0

Component : API GATEWAY

Resolution

The issue only happens for conditional query (has IF EXISTS clause in sql),  work around is to remove the the "if exists" condition for the query. 

More details on this are as follows:

The consistency level configured in Gateway works as expected in general, but there are some exceptions based on DataStax's implementation.

One such condition is when the query triggers a lightweight transaction - these include INSERT, UPDATE or DELETE statements with an IF condition.

Lightweight transactions are implemented in Cassandra using the Paxos consensus protocol, which has multiple phases. The driver documentation describes the consistency-phase relation in this way: "the serial consistency level defines the consistency level of the serial phase (or "paxos" phase) while the normal consistency level defines the consistency for the 'learn' phase". 

So during lightweight transactions, one of the serial consistency levels (SERIAL, LOCAL_SERIAL) shall be used for the "paxos" phase, and the configured consistency level will be used for the "learn phase".

 

Refer to DataStax documentation for more details:

https://docs.datastax.com/en/drivers/java/3.6/com/datastax/driver/core/Statement.html#setSerialConsistencyLevel-com.datastax.driver.core.ConsistencyLevel-

If Debugging is required in datastax it can be enabled by log.properties logger of,

 com.datastax.driver.core.level = FINE

This will require modifying the logsink to include ALL OR FINE level messages.