Troubleshooting VCHA failover failed due to large amount of stellar.db files

Products

• VMware vCenter Server
• VMware vCenter Server 7.0

Issue/Introduction

Overview:

This knowledge-based article outlines the steps to diagnose and resolve issues when a VCHA failover fails, particularly when file replication is stuck in progress.

Symptoms:

When a manual failover is attempted in a VCHA environment, the failover process fails due to file replication being stuck in progress. This article provides guidance on identifying the root cause and addressing this issue effectively.

Environment

VMware vCenter Server 7.0.3

Cause

Rsync timeout happend due to large amount of stellar.db file and there was bloated observervalues table.

Resolution

Cleaning up FDSize metric records from observervalues table and disabling FDSize metric would be helpful.

 

1) Open Sqlite db:
sqlite3 <stellar.db path>
sqlite3 /storage/db/stellar.db

2) disable metric
update metrics set status = 0 where name ="FDSize"; 

3) clean up observalues table
delete from observervalues where metricname ="FDSize";

4) restart observability service using below command - 
service-control --restart observability OR systemctl restart observability

5) Retry the Failover step (rsync).