ERROR: WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():775: Will never receive state. Need to abort.

Products

• VMware Integrated OpenStack

Issue/Introduction

• DB is in a bad state
• One of the database pods is in CrashLoopBackOff similar to below
  openstack   mariadb-server-0   0/1     CrashLoopBackOff   56         2d17h    <pod ip address>   controller-##########      <none>           <none>
  
• Pod logs will show similar to below:
  OpenStack-Helm Mariadb - INFO - b'2025-01-01 11:35:11 140383673632512 [Warning] WSREP: 0.0 (mariadb-server-1.mariadb-discovery.openstack.svc.cluster.local): State transfer to 2.0 (mariadb-server-0.mariadb-discovery.openstack.svc.cluster.local) failed: -255 (Unknown error 255)'
OpenStack-Helm Mariadb - INFO - b'2025-01-01 11:35:11 140383673632512 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():775: Will never receive state. Need to abort.'

Environment

7.x

Cause

The node is unable to join the Galera Cluster due to an unknown error during state transfer attempt.

Resolution

1. Identify the reboot node
   #osctl get cm mariadb1-mariadb-state -oyaml

...
   metadata:
     annotations:
       openstackhelm.openstack.org/cluster.state: reboot
       openstackhelm.openstack.org/leader.expiry: "2022-04-06T12:49:54.353900Z"
       openstackhelm.openstack.org/leader.node: mariadb-server-1
       openstackhelm.openstack.org/reboot.node: mariadb-server-0
...

2. Identify the leader node (see above)
   
   
3. If the leader node is one of the working nodes, delete the non-working node
   #osdel pod mariadb-server-x (where x is the number of the non-working node)
   
   
4. The deleted pod will restart and then rejoin the cluster and sync state.

Additional Information

If the delete pod does not start back up and come to a Running status, contact Broadcom Support.