terminating walreceiver due to timeout
Article ID: 407762
Updated On:
Products
VMware Tanzu Data Suite
VMware Tanzu Greenplum
VMware Tanzu Greenplum / Gemfire
Issue/Introduction
+ You might be noticing below detailed errors in the segment mirror logs and primary log pairs.
+ Sometimes no action required from you and DOWN mirrors marked by FTS might marked back UP and in SYNC with its primary.
+ Sometimes you might need to explicitly run the gprecoverseg for incremental recovery to mark the DOWN mirror back UP and in SYNC.
Mirror Logs
2025-07-05 00:14:14.888574 PDT,,,p13297,th440448128,,,,0,,,seg4,,,,,"LOG","00000","restartpoint complete: wrote 1634 buffers (40.9%); 0 transaction log file(s) added, 0 removed, 8 recycled; write=149.212 s, sync=8.595 s, total=182.615 s; sync files=2376, longest=8.578 s, average=0.003 s",,,,,,,0,,"xlog.c",8943,
2025-07-05 00:14:14.888813 PDT,,,p13297,th440448128,,,,0,,,seg4,,,,,"LOG","00000","recovery restart point at 4806/9803D5D8","last completed transaction was at log time 2025-07-05 00:12:57.8895-07",,,,,,0,,"xlog.c",9990,
2025-07-05 00:14:14.890685 PDT,,,p13297,th440448128,,,,0,,,seg4,,,,,"LOG","00000","restartpoint starting: xlog",,,,,,,0,,"xlog.c",8870,
2025-07-05 00:14:22.243077 PDT,,,p13299,th440448128,,,,0,,,seg4,,,,,"ERROR","XX000","terminating walreceiver due to timeout",,,,,,,0,,"walreceiver.c",555,"Stack trace:
1 0xbe57e0 postgres errstart (elog.c:567)
2 0xa6aae5 postgres WalReceiverMain (walreceiver.c:555)
3 0x7fac06 postgres AuxiliaryProcessMain (bootstrap.c:362)
4 0xa3c4a4 postgres <symbol not found> (postmaster.c:5778)
5 0xa3ddb5 postgres <symbol not found> (postmaster.c:2149)
6 0x7fa61a23ebf0 libc.so.6 <symbol not found> + 0x1a23ebf0
7 0x7fa61a304769 libc.so.6 __select + 0xb9
8 0xa3efc5 postgres <symbol not found> (postmaster.c:1888)
9 0xa4124c postgres PostmasterMain (discriminator 5)
10 0x73547a postgres main (main.c:178)
11 0x7fa61a2295d0 libc.so.6 <symbol not found> + 0x1a2295d0
12 0x7fa61a229680 libc.so.6 __libc_start_main + 0x80
13 0x741315 postgres _start + 0x25
"Primary Logs
2025-07-05 00:14:29.977228 PDT,"gpadmin",,p1477277,th-967019392,"10.10.1.102","60062",2025-07-05 00:14:22 PDT,0,,,seg4,,,,,"LOG","00000","standby ""gp_walreceiver"" is now the synchronous standby with priority 1",,,,,,,0,,"syncrep.c",594,
2025-07-05 00:14:57.027790 PDT,"gpadmin",,p1477416,th-967019392,"10.10.1.11","19728",2025-07-05 00:14:57 PDT,0,,,seg4,,,,,"LOG","00000","turning off synchronous wal replication due to FTS request",,,,,,,0,,"ftsmessagehandler.c",312,
2025-07-05 00:15:19.049871 PDT,"gpadmin",,p1477541,th-967019392,"10.10.1.11","48532",2025-07-05 00:15:19 PDT,0,,,seg4,,,,,"LOG","00000","turning off synchronous wal replication due to FTS request",,,,,,,0,,"ftsmessagehandler.c",312,
2025-07-05 00:15:41.004280 PDT,"gpadmin",,p1477740,th-967019392,"10.10.1.11","49810",2025-07-05 00:15:41 PDT,0,,,seg4,,,,,"LOG","00000","turning off synchronous wal replication due to FTS request",,,,,,,0,,"ftsmessagehandler.c",312,
2025-07-05 00:15:47.440675 PDT,"gpadmin",,p1477566,th-967019392,"10.10.1.102","62838",2025-07-05 00:15:30 PDT,0,,,seg4,,,,,"ERROR","55006","replication slot ""internal_wal_replication_slot"" is already active",,,,,,,0,,"slot.c",361,
2025-07-05 00:16:03.003497 PDT,"gpadmin",,p1477856,th-967019392,"10.10.1.11","38036",2025-07-05 00:16:03 PDT,0,,,seg4,,,,,"LOG","00000","turning off synchronous wal replication due to FTS request",,,,,,,0,,"ftsmessagehandler.c",312,
2025-07-05 00:16:06.026085 PDT,"gpadmin",,p1477416,th-967019392,"10.10.1.11","19728",2025-07-05 00:14:57 PDT,0,,,seg4,,,,,"LOG","00000","signaling configuration reload: setting synchronous_standby_names to ''",,,,,,,0,,"gp_replication.c",619,
2025-07-05 00:16:06.026250 PDT,,,p12734,th-967019392,,,,0,,,seg4,,,,,"LOG","00000","received SIGHUP, reloading configuration files",,,,,,,0,,"postmaster.c",2975,
2025-07-05 00:16:06.068719 PDT,,,p12734,th-967019392,,,,0,,,seg4,,,,,"LOG","00000","parameter ""synchronous_standby_names"" changed to """"",,,,,,,0,,"guc-file.l",422,
Log file - gpdb-2025-07-05_001606.csv
2025-07-05 00:16:06.071407 PDT,,,p13453,th-967019392,,,,0,,,seg4,,,,,"LOG","00000","ic-proxy: received signal 1",,,,,,,0,,"ic_proxy_main.c",400,
2025-07-05 00:16:08.123468 PDT,"gpadmin",,p1477541,th-967019392,"10.10.1.11","48532",2025-07-05 00:15:19 PDT,0,,,seg4,,,,,"LOG","00000","signaling configuration reload: setting synchronous_standby_names to ''",,,,,,,0,,"gp_replication.c",619,
2025-07-05 00:16:08.123510 PDT,,,p13448,th-967019392,,,,0,,,seg4,,,,,"LOG","00000","checkpoint complete: wrote 264 buffers (6.6%); 0 transaction log file(s) added, 0 removed, 8 recycled; write=27.126 s, sync=2.841 s, total=210.803 s; sync files=83, longest=2.840 s, average=0.034 s",,,,,,,0,,"xlog.c",8959,
2025-07-05 00:16:08.123554 PDT,,,p12734,th-967019392,,,,0,,,seg4,,,,,"LOG","00000","received SIGHUP, reloading configuration files",,,,,,,0,,"postmaster.c",2975,
2025-07-05 00:16:08.123584 PDT,"gpadmin",,p1477541,th-967019392,"10.10.1.11","48532",2025-07-05 00:15:19 PDT,0,,,seg4,,,,,"LOG","08006","could not send data to client: Broken pipe",,,,,,,0,,"pqcomm.c",1593,
2025-07-05 00:16:08.123610 PDT,"gpadmin",,p1477541,th-967019392,"10.10.1.11","48532",2025-07-05 00:15:19 PDT,0,,,seg4,,,,,"FATAL","08006","connection to client lost",,,,,,,0,,"postgres.c",3846,
Log file - gpdb-2025-07-05_001608.csv
025-07-05 00:16:08.124915 PDT,,,p13453,th-967019392,,,,0,,,seg4,,,,,"LOG","00000","ic-proxy: received signal 1",,,,,,,0,,"ic_proxy_main.c",400,
2025-07-05 00:16:08.142902 PDT,"gpadmin",,p1477850,th-967019392,"10.10.1.102","48084",2025-07-05 00:15:47 PDT,0,,,seg4,,,,,"WARNING","42704","FTSReplicationStatus ""gp_walreceiver"" does not exist",,,,,,,0,,"gp_replication.c",200,
2025-07-05 00:16:08.147192 PDT,"gpadmin",,p1477850,th-967019392,"10.10.1.102","48084",2025-07-05 00:15:47 PDT,0,,,seg4,,,,,"LOG","00000","standby ""gp_walreceiver"" is now the synchronous standby with priority 1",,,,,,,0,,"syncrep.c",594,
2025-07-05 00:16:09.131731 PDT,"gpadmin",,p1477740,th-967019392,"10.10.1.11","49810",2025-07-05 00:15:41 PDT,0,,,seg4,,,,,"LOG","00000","signaling configuration reload: setting synchronous_standby_names to ''",,,,,,,0,,"gp_replication.c",619,
2025-07-05 00:16:09.131772 PDT,,,p12734,th-967019392,,,,0,,,seg4,,,,,"LOG","00000","received SIGHUP, reloading configuration files",,,,,,,0,,"postmaster.c",2975,
Environment
Greenplum All versions.
Cause
No RCA found specifically. There could be number of reasons but all those leading to primary not able to communicate/sync to mirror and Fault Tolerance System(FTS) mark the mirror DOWN after trying for duration (controlled by GUC wal_receiver_timeout - default 60 seconds).
Resolution
Since there is no specific RCA found for this behavior no permanent fix for these scenarios but there are workarounds that might help as it resolved the issues in the past.
Workaround -
Example listed below and you might want to increase the default from 60 to either 90,120 etc
gpconfig -c wal_receiver_timeout -v 120sUpdate the configuration without actual restart of the database.
gpstop -uFeedback
Yes
No
Powered by
