Postgres Node in 'Unknown' State After Tanzu Hub Upgrade

Products

• VMware Tanzu Platform - Hub

Issue/Introduction

After upgrading Tanzu Hub, the Postgres 3-node/Patroni cluster encountered issues where two members were working (one leader and one streaming/standby), but the third node was in an 'unknown' state and logging the following errors.

Logs from postgresql

level-error msg="Error opening connection to database" err="error querying postgresql version: dial tcp ###.#.#.#:5432: connect: connection refused"

If a database node is not in sync, there is a risk of a WAL replication issue later.

Resolution

Rerun the Tanzu Hub errand, SSH to the primary postgres pod and execute the below command to check the HA nodes status and state:

patronictl list  

If the third replica remains in an unknown state, it can be reinitialized using the command:

patronictl reinit <cluster> <member> 

(where <cluster> is usually postgresql and <member> is the name of the unknown node, e.g., postgresql-X)