pg_xlog filled up for only one segment

Products

• VMware Tanzu Greenplum

Issue/Introduction

Noticing that pg_xlog of a mirror segment has filled up and taking lot more space than its primary and other mirror segments.
 

[gpadmin@dtord03gpfs105p mirror]$ du -hs gpseg3/pg_xlog 2.1T gpseg3/pg_xlog
[gpadmin@dtord03gpfs105p mirror]$ du -hs gpseg214/pg_xlog 1.5G gpseg214/pg_xlog

As you can see gpseg3 is taking 2.1 T while gpseg214 is taking only 1.5G.

[gpadmin@dtord03gpfs101p primary]$ du -hs gpseg3/pg_xlog 1.5G gpseg3/pg_xlog

The corresponding primary segment of gpseg3 is also taking 1.5 G

Environment

Product Version: 6.19

Resolution

Issue caused by bug. fixed in 6.20.0

32098, 32079 : If the checkpointer process failed to open segment 1, the segment number was incorrectly reset to 0.
This could cause an infinite loop of retries and failures and lead to the pg_xlog file growing in size with the checkpointer process hanging. The problem was resolved by ensuring that the segment number is not reset to 0 in this situation
 

To resolve this issue, we have had to stop the segment

pg_ctl -D <segment_directory> stop -m immediate
e.g.
pg_ctl -D /data1/mirror/gpseg3 stop -m immediate

 

Then we recovered the segment with gprecoverseg. Incremental recovery has worked, but sometimes a full recovery may be needed.

This bug is fixed in 6.19.4 and above, so please plan to upgrade when you can to avoid this again.