Healthwatch pxc-mysql vms in cluster failing

Products

• VMware Tanzu Kubernetes Grid Integrated Edition
• VMware Tanzu Application Service for VMs

Issue/Introduction

When there is a pxc-mysql vm cluster (3+ vms) in the Healthwatch deployment, it may sometimes enter a failing state.

When the cluster loses quorum and fails, you may need to bootstrap it to fix it. This can also results in the Grafana VM failing, as it uses pxc-mysql database.

Environment

Healthwatch pxc-mysql cluster.

Cause

A cluster loses quorum when fewer than half the nodes can communicate with each other for longer than the configured grace period. A review of the MySQL logs should help identify where exactly the failure occurred.

Resolution

It is important to verify that a bootstrap is needed, before taking any actions towards bootstrapping. Please raise a case with Broadcom support if you have any uncertainty. 

To fix the issue with pxc-mysql cluster you can do the following options:

1. If the VMs themselves are in running state but have failing monit processes, you can use bootstrap errand:

bosh -d p-healthwatch2-e30c541cea8ea81634e6 run-errand bootstrap

2. Otherwise the manual bootstrap is needed to fix the cluster. The steps to run a manual bootstrap is available here Manual Bootstrap