How to reset etcd data of a multi-master VMware Tanzu Kubernetes Grid Integrated Edition cluster
search cancel

How to reset etcd data of a multi-master VMware Tanzu Kubernetes Grid Integrated Edition cluster

book

Article ID: 298634

calendar_today

Updated On: 04-20-2024

Products

VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

How to reset etcd data of a multi-master VMware Tanzu Kubernetes Grid Integrated Edition cluster.

WARNING: The procedure in this KB will wipe out the current etcd data. This is intended only for scenarios where you deliberately want to delete the current etcd data to bootstrap a new etcd cluster.

Typical scenario where this procedure can be used is when performing a BBR Restore of a multi-master PKS cluster and you ran into an etcd startup problem with the redeployment step. The current etcd data during the redeployment step in this scenario is not important because it will restored during the Restore step using the backup artifact. You can only proceed with a restore once the VM's are in a healthy state but because there's a problem with etcd startup, you are blocked and cannot proceed with the restore. This procedure would help in this scenario.

Environment

Product Version: 1.7

Resolution

For each master VM:

1. Log in via bosh ssh into the master VM and become root
$ bosh ssh master/xxx
$ sudo -i

2. Make sure all jobs are running except for etcd and kube-apiserver
$ monit start all
Wait a few moments and monitor the jobs state by running monit summary. When all processes except etcd and kube-apiserver (and flanneld) are in a running state, stop etcd and kube-apiserver:
$ monit stop etcd
$ monit stop kube-apiserver

3. Delete the etcd member directory:
$ rm -r /var/vcap/store/etcd/member
Afterwards, run the following to make sure each master's etcd and kube-apiserver are not running.
$ bosh -d <cluster deployment name> is --ps
Then, ssh into master/0 and start etcd and kube-apiserver.
$ monit start etcd
$ monit start kube-apiserver
Repeat the above etcd and kube-apiserver start up on the remaining masters.