How to recover inconsistent Postgresql slave database?

book

Article ID: 193997

calendar_today

Updated On:

Products

CA API Developer Portal

Issue/Introduction

We have a Portal instance using the internal Postgresql database in the pre-production environment. The slave database seems to be in an inconsistent state and is unsuccessfully trying to recover. 

The logs are showing following lines:

[email protected]    | LOG:  redo starts at 3/[email protected]    | LOG:  consistent recovery state reached at 3/[email protected]    | LOG:  record with incorrect prev-link 0/1 at 3/[email protected]    | LOG:  database system is ready to accept read only [email protected]    | LOG:  started streaming WAL from primary at 3/84000000 on timeline [email protected]    | FATAL:  could not receive data from WAL stream: ERROR:  requested WAL segment 000000010000000300000084 has already been removed

What is the proper procedure to rebuild or recover the slave database and restart replication?

Environment

Release : 4.x

Component : API PORTAL

Resolution

You can recover the portal_portaldb-slave container by executing the following steps

Make sure you have a valid database backup 

https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-api-management/api-developer-portal/4-4/set-up-and-maintenance/maintain-the-api-developer-portal/back-up-and-restore-internal-database.html

Remove the docker stack  :

       docker stack rm portal


List the docker persistent volumes :

       docker volume ls

Remove the slave persistent volume "portal_database-postgres-slave-volume" :

       docker volume rm portal_database-postgres-slave-volume

Restart the portal by running :

       portal.sh

Verify if all containers are starting :

       docker service ls