Standby master or mirror would show "FATAL the database system is in recovery mode" while gpstate shows the cluster is in sync in Pivotal Greenplum

Products

• VMware Tanzu Greenplum

Issue/Introduction

Sometime, we may notice below FATAL error message in the standby master or mirror as below: 

2020-03-17 00:23:01.276048 EDT,"gpadmin","template1",p5024,th-346093440,"172.28.8.2","7229",2020-03-17 00:23:01 EDT,0,,,seg-1,,,,,"FATAL","57M02","the database system is in recovery mode","last replayed record at 0/3C000000
- VERSION: PostgreSQL 9.4.24 (Greenplum Database 6.4.0 build commit:564b89a8c6bef5e329a59f39dac438b13d9cb3fa) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Feb 11 2020 20:35:16",,,,,,0,,"postmaster.c",2578,
2020-03-17 00:23:01.292466 EDT,"gpadmin","gpadmin",p5026,th-346093440,"172.28.8.2","7231",2020-03-17 00:23:01 EDT,0,,,seg-1,,,,,"FATAL","57M02","the database system is in recovery mode","last replayed record at 0/3C000000
- VERSION: PostgreSQL 9.4.24 (Greenplum Database 6.4.0 build commit:564b89a8c6bef5e329a59f39dac438b13d9cb3fa) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Feb 11 2020 20:35:16",,,,,,0,,"postmaster.c",2578,

Environment

Product Version: 6.4

Resolution

This error message is a normal message for standby master or mirror while syncing. The reason of that is while standby or mirror is syncing, it would be in the recovery mode. If we connect them via utility mode at that time, this FATAL error message will pop out. Unfortunately, we have a lot of process & tools would connect to standby/mirror via utility mode when GPDB is in normal syncing status, which makes this FATAL message shows up frequently in the logs.

As long as the gpstate output shows the result is in-sync, we can trust that result and safely ignore this FATAL error message.