Large "Lag in MB" is observed for vcf-fleet-lcm-db in VCF 9.1

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

In a VCF 9.1 environment, when the environment recovers from an outage, some vcf-fleet-lcm-db members may show a large "Lag in MB" value. This can be verified by running the following command:

kubectl -n vcf-fleet-lcm exec vcf-fleet-lcm-db-0 -- patronictl list

+ Cluster: vcf-fleet-lcm-db (###################) -+---------------------+----+-----------+
| Member             | Host         | Role         | State               | TL | Lag in MB |
+--------------------+--------------+--------------+---------------------+----+-----------+
| vcf-fleet-lcm-db-0 |  IP_ADDRESS  | Sync Standby | streaming           |  2 |         0 |
| vcf-fleet-lcm-db-1 |  IP_ADDRESS  | Replica      | in archive recovery |  2 |      1### |
| vcf-fleet-lcm-db-2 |  IP_ADDRESS  | Leader       | running             |  2 |           |
+--------------------+--------------+--------------+---------------------+----+-----------+

Environment

VCF 9.1

Cause

A rare condition prevents the synchronization from the Leader node to the Replica or Sync Standby nodes.

Resolution

Run the following command to reinitialize the member reporting the large "Lag in MB" value. 

Note: In the example above, "vcf-fleet-lcm-db-1" has the large lag and "vcf-fleet-lcm-db-2" is the healthy leader. The command should be executed from a healthy member against the specific member requiring reinitialization.

kubectl exec vcf-fleet-lcm-db-2 -n vcf-fleet-lcm -- patronictl reinit vcf-fleet-lcm-db vcf-fleet-lcm-db-1 --force

Additional Information

How to remediate an unhealthy postgres cluster

VCF Identity Broker encountered an issue during authentication