After switching to standby master a GPCC function gpcc_schema.get_querytext() would report error
Article ID: 407812
Updated On:
Products
Issue/Introduction
In Greenplum6, after switching to a standby master and start GPCC there on the new master, pg_log can report error with running gpmon query:
2025-08-11 00:00:28.109631 <TZ>,"gpmon","gpperfmon",p275747,th-1090357376,"<IP Addr>","15600",2025-08-11 00:00:24 <TZ>,0,con205779,cmd6,seg-1,,dx3714467,,sx1,"ERROR","XX000","function ""get_querytext"" error fetching next item from iterator (plpython.c:5043)","KeyError: 'MASTER_DATA_DIRECTORY'",,,,"Traceback (most recent call last):
PL/Python function ""get_querytext"", line 39, in <module>
PL/Python function ""get_querytext""","SELECT * FROM gpcc_schema.get_querytext('', 1000) AS file",0,,"plpython.c",5043,"Stack trace:
1 0x96657b postgres errstart (elog.c:521)
2 0x7f189b899eda plpython.so <symbol not found> + 0x9b899eda
3 0x7f189b8a1ada plpython.so <symbol not found> + 0x9b8a1ada
4 0x7f189b8a3074 plpython.so plpython_call_handler + 0x104
5 0x6aa741 postgres ExecMakeTableFunctionResult (execQual.c:2174)
6 0x6c65ea postgres <symbol not found> (nodeFunctionscan.c:82)
7 0x6a15e3 postgres ExecProcNode (execProcnode.c:994)
8 0x69e68c postgres <symbol not found> (execMain.c:2900)
9 0x69ed32 postgres ExecutorRun (execMain.c:912)
10 0x843729 postgres <symbol not found> (pquery.c:1165)
11 0x8453f5 postgres PortalRun (pquery.c:988)
12 0x83fec6 postgres <symbol not found> (postgres.c:1783)
13 0x8420e1 postgres PostgresMain (postgres.c:4975)
14 0x7e1565 postgres <symbol not found> (postmaster.c:6419)
15 0x7e3d3a postgres PostmasterMain (postmaster.c:1533)
16 0x4cdf17 postgres main (main.c:206)
17 0x7f18ba52e3d5 libc.so.6 __libc_start_main + 0xf5
18 0x4ce4cc postgres <symbol not found> + 0x4ce4cc
"
Environment
GPDB 6.28 + GPCC 6.14
Cause
When standby master was initiated, gpinitstandby performed the following:
1) ssh into the standby host
2) Run pg_ctl start to launch the standby
3) At such moment, strings /proc/<postmaster pid>/environ | grep MASTER_DATA_DIRECTORY will output nothing which indicate that the standby master server process was not set with such required variable. (strings converts the binary-ish content into readable text, showing each environment variable line by line)
The root cause is that each ssh session starts with a fresh environment, if not set in advance the Linux kernel will not pass env var MASTER_DATA_DIRECTORY to the standby master postmaster process, which further result in the mentioned query failure.
Resolution
Temporary Workaround
- source greenplum_path.sh and restart database after switching to standby master
Resolution
This will be fixed in higher version Greenplum 6X.
Feedback
