Healthwatch pxc-mysql vms in cluster failing
Article ID: 389352
Updated On:
Products
Issue/Introduction
When there is a pxc-mysql vm cluster (3+ vms) in the Healthwatch deployment, it may sometimes enter a failing state.
When the cluster loses quorum and fails, you may need to bootstrap it to fix it. This can also results in the Grafana VM failing, as it uses pxc-mysql database.
Environment
Healthwatch pxc-mysql cluster.
Cause
A cluster loses quorum when fewer than half the nodes can communicate with each other for longer than the configured grace period. A review of the MySQL logs should help identify where exactly the failure occurred.
Resolution
It is important to verify that a bootstrap is needed, before taking any actions towards bootstrapping. Please raise a case with Broadcom support if you have any uncertainty.
To fix the issue with pxc-mysql cluster you can do the following options:
1. If the VMs themselves are in running state but have failing monit processes, you can use bootstrap errand:
bosh -d p-healthwatch2-##### run-errand bootstrap
2. Otherwise the manual bootstrap is needed to fix the cluster. The steps to run a manual bootstrap is available here Manual Bootstrap
Feedback
