Avi Controller UI/API inaccessible - Postgres Replication Failure

Products

• VMware Avi Load Balancer

Issue/Introduction

The controller UI/API may become inaccessible and return an error "error": "Bad Service".  Some of the cluster nodes may be stuck in "Starting state"



Run below command in CLI to check the nodes status.

show cluster nodes

Environment

This issue may occur on all cloud environments

Cause

The Postgres replication may fail due to

• Postgres db (Config or Metrics) got corrupted
• Postgres did not get initialized correctly on one of the nodes
• Replication not complete flag is present, when the pg_main_replication_not_complete or pg_metrics_replication_not_complete is still present in /var/lib/avi/etc/
  

Check if replication_not_complete file is written.

sudo -i
find / -name *replication_not* -type f 2>&1 | grep -v find

Resolution

Contact Broadcom Support for further assistance on this issue.

https://knowledge.broadcom.com/external/article/405686/how-to-create-a-wolken-case-for-avi-prod.html