Query appears to have failed at segment with OOM message and master log showing query is successful

Products

• VMware Tanzu Data Suite

Issue/Introduction

Master log:

2025-05-19 15:30:37.107304 UTC,"user1","DB1",p156559,th-707958656,"9.10.11.12","46190",2025-05-19 15:18:21 UTC,0,con4026734,cmd34,seg-1,,,,,"LOG","00000","duration: 735383.233 ms",,,,,,"select * from schema1.UDF ( ... );",0,,"postgres.c",1927,

Segment log:

2025-05-19 15:30:37.269838 UTC,"user1","DB1",p208408,th-247150464,"9.10.12.13","48164",2025-05-19 15:20:25 UTC,0,con4026734,,seg776,,,,,"LOG","00000","One or more query
 execution processes ran out of memory on this segment. Logging memory usage.",,,,,,,0,,,,

 

Cause

In this scenario, con4026734 did not fail on the segment based on the following log record. This is a LOG message, and not an ERROR. 

UTC,0,con4026734,,seg776,,,,,"LOG","00000","One or more query execution processes ran out of memory on this segment. Logging memory usage

This is basically indicating that some process(es) ran out of memory, and so the active process is going to log its memory usage for the purpose of helping with debugging.

 

Following is a more detailed explanation:

 

The message of concern (above) on the segment side is a LOG message, LOG is not an ERROR, and so It is normal.

When Out Of Memory happens, it might not only be because of a single session (process). So the original design has been put in place to dump the segment's memory usage in real time. 

At around the same time, other workers in the same segment might also dump their memory usage. This design is so that when OOM happen, the full LOG would continue to be used to re-build the image around the point of OOM condition.

Resolution

No direct action is required to address the message "One or more query execution processes ran out of memory on this segment".

There is however a considerable possibility there could be actual Out Of Memory errors around the same time by other processes, that woudl involve separate course of troubleshooting.