VIP Authentication Hub Hazelcast replication issue
Article ID: 256599
Updated On:
Products
Issue/Introduction
We deployed the M9 version of Authentication Hub in our dev region, then began experiencing "invalid x-flow-state header" errors. The logs returned messages such as:
No cache entry found for flowStateId: <flow_id>
No data found for flowStateId: <flow_id>
Could not process the x-flow-state header, returning BadRequest.
The errors occurred at the following endpoints:
SelectedFactor/
factor authentication endpoints
The authentication flow succeeded with multiple submissions. The failed and successful submissions shared the same flow state id. The fail/success rate was a nearly even 50/50 split, and we had two instances. Thus we believe this was a replication issue.
Environment
Release : Oct.03
Cause
Looking at HC logs We can see that the deployment of the HC was not completed successfully. It look like the deployment was not able to create the hazelcast-cluster-role (ClusterRole object - https://raw.githubusercontent.com/hazelcast/hazelcast/master/kubernetes-rbac.yaml)
Without this object HC discovery (which allows it to find the other HC pods) will not work and HC thinks it's in standalone more - i.e. no replication.
You can see that in the HC log:
Kubernetes API access is forbidden! Starting standalone. To use Hazelcast Kubernetes discovery, configure the required RBAC. For 'default' service account in 'default' namespace execute: `kubectl apply -f https://raw.githubusercontent.com/hazelcast/hazelcast/master/kubernetes-rbac.yaml`
Resolution
After Oct.04( Releasing 12/21/2022) is deployed, These are the choices customer may have:
Option 1. Explicitly disable HC RBAC via hazelcast-enterprise.rbac.create=false. AH will automatically bootstrap HC pods to come up in DNS_LOOKUP mode.
this will be shown in HC log:
2022-12-13 19:26:34,026 [ INFO] [main] [c.h.s.d.i.DiscoveryService]: [10.20.2.44]:5702 [dev] [5.1.3] Kubernetes Discovery activated with mode: DNS_LOOKUP
Option 2. Leave HC RBAC enabled (hazelcast-enterprise.rbac.create=true). AH will automatically bootstrap HC in KUBERNETES_API mode
this will be shown in HC log:
2022-12-13 19:22:17,368 [ INFO] [main] [c.h.s.d.i.DiscoveryService]: [10.20.2.43]:5702 [dev] [5.1.3] Kubernetes Discovery activated with mode: KUBERNETES_API
HC prefers KUBERNETES_API discovery mode (https://docs.hazelcast.com/hazelcast/5.1/deploy/configuring-kubernetes#discovering-members), but right now we don't have empirical evidence to suggest this is required. If you're trying to run in the least privileged mode, then go with #1. We can always switch later and if it becomes required, that will be the reason for requiring HC RBAC.
Whatever option you decide to go with, after upgrading to Oct.04 please scale down HC pods to 0, then scale up to 3.
Feedback
