VMware Tanzu Greenplum: PANIC: resource group wait queue is corrupted

Products

• VMware Tanzu Greenplum

Issue/Introduction

This issue can occur when using resource groups. The error message is:

PANIC: resource group wait queue is corrupted

The corefile backtrace will show one of the following:

(gdb) bt
#0  0x00007fed81dc9387 in raise () from /data/logs/269771/packcore-core-postgres-173172-1613007195/lib64/libc.so.6
#1  0x00007fed81dcaa78 in abort () from /data/logs/269771/packcore-core-postgres-173172-1613007195/lib64/libc.so.6
#2  0x0000000000bf0b83 in errfinish (dummy=dummy@entry=0) at elog.c:763
#3  0x0000000000bf294c in elog_finish (elevel=elevel@entry=22, fmt=fmt@entry=0x11aed48 "resource group wait queue is corrupted") at elog.c:1749
#4  0x0000000000c32f83 in groupWaitQueueValidate (group=<optimized out>) at resgroup.c:3502
#5  groupWaitQueuePush (proc=0x7fed75549e00, group=<optimized out>) at resgroup.c:3547
#6  groupAcquireSlot (pGroupInfo=pGroupInfo@entry=0x7ffd55b44ad0, isMoveQuery=isMoveQuery@entry=0 '\000') at resgroup.c:1918
#7  0x0000000000c3456e in AssignResGroupOnMaster () at resgroup.c:2649
#8  0x000000000072e175 in StartTransaction () at xact.c:2587
#9  0x0000000000731b95 in StartTransactionCommand () at xact.c:3496
#10 0x0000000000a871c9 in start_xact_command () at postgres.c:3206
#11 exec_parse_message (query_string=query_string@entry=0x2905fd9 "SELECT EXISTS (\nSELECT 1\nFROM information_schema.tables\nWHERE table_name = $1)", stmt_name=stmt_name@entry=0x2905fd8 "", paramTypes=<optimized out>, paramTypes@entry=0x2906848, numParams=<optimized out>, 
    numParams@entry=1) at postgres.c:1963
#12 0x0000000000a89409 in PostgresMain (argc=<optimized out>, argv=argv@entry=0x291dc10, dbname=<optimized out>, username=<optimized out>) at postgres.c:5477
#13 0x00000000006b0f25 in BackendRun (port=0x2915fa0) at postmaster.c:4811
#14 BackendStartup (port=0x2915fa0) at postmaster.c:4468
#15 ServerLoop () at postmaster.c:1948
#16 0x0000000000a0fc72 in PostmasterMain (argc=argc@entry=6, argv=argv@entry=0x28e3a00) at postmaster.c:1518
#17 0x00000000006b4cf1 in main (argc=6, argv=0x28e3a00) at main.c:245

OR:

(gdb) bt
#0  0x00007fe33d0e9377 in raise () from /data/logs/280302/packcore-core.postgres.40736.1622510422.6.8500.8500/lib64/libc.so.6
#1  0x00007fe33d0eaa68 in abort () from /data/logs/280302/packcore-core.postgres.40736.1622510422.6.8500.8500/lib64/libc.so.6
#2  0x000000000096532d in errfinish (dummy=dummy@entry=0) at elog.c:703
#3  0x0000000000967480 in elog_finish (elevel=elevel@entry=22, fmt=fmt@entry=0xd03660 "resource group wait queue is corrupted") at elog.c:1482
#4  0x00000000009a0d37 in groupWaitProcValidate (head=<optimized out>, proc=0x7fe32d74bcc0) at resgroup.c:3418
#5  groupWaitQueuePop (group=0x7fe32d668038) at resgroup.c:3466
#6  wakeupSlots (group=group@entry=0x7fe32d668038, grant=1 '\001') at resgroup.c:2186
#7  0x00000000009a31d1 in groupReleaseSlot (slot=<optimized out>, group=<optimized out>) at resgroup.c:2386
#8  UnassignResGroup () at resgroup.c:2687
#9  0x000000000051e533 in CommitTransaction () at xact.c:2885
#10 0x000000000051ef85 in CommitTransactionCommand () at xact.c:3689
#11 0x000000000083a503 in finish_xact_command () at postgres.c:3228
#12 0x000000000084088d in finish_xact_command () at postgres.c:3219
#13 PostgresMain (argc=<optimized out>, argv=argv@entry=0x13bb328, dbname=<optimized out>, username=<optimized out>) at postgres.c:5431
#14 0x00000000007e0ad5 in BackendRun (port=0x13df490) at postmaster.c:6732
#15 BackendStartup (port=0x13df490) at postmaster.c:6406
#16 ServerLoop () at postmaster.c:2444
#17 0x00000000007e32aa in PostmasterMain (argc=argc@entry=15, argv=argv@entry=0x1368ca0) at postmaster.c:1528
#18 0x00000000004cdbb7 in main (argc=15, argv=0x1368ca0) at main.c:206

The cause of the issue is that when processes are waiting for resource group slots, they could be added into the queue, which will cause the corruption of the queue.
Code changes have been made in 5.28.7 and 6.15.0 to change this behavior.

Environment

Product Version: 5.28

Resolution

Code fix is available in:

• GPDB 5.28.7 and above
• GPDB 6.15.0 and above