Healthwatch pxc-mysql vms in cluster failing

Products

• VMware Tanzu Kubernetes Grid Integrated Edition
• VMware Tanzu Application Service for VMs

Issue/Introduction

When there is a pxc-mysql vm cluster ( 3 vms) in Healthwatch deployment sometimes it may fail. When the cluster loses quorum and fails you may need to bootstrap it to fix it. This results in Grafana vm failure as it uses pxc-mysql database.

Environment

Healthwatch pxc-mysql cluster.

Resolution

To fix the issue with pxc-mysql cluster you can do the following options:

1. If vms are in running state but failing processes you can use bootstrap errand:

bosh -d p-healthwatch2-e30c541cea8ea81634e6 run-errand bootstrap

2. Otherwise  the manual bootstrap is needed to fix the cluster. The steps to run a manual bootstrap is available here Manual Bootstrap