segment failure due to security software. error "Could not open file ""pg_distributedlog/xxxx"": Operation not permitted.

Products

• VMware Tanzu Data Suite

Issue/Introduction

If we met segment failure (mirror or primary), from the pg log, we found below message:

"FATAL","42501","could not access status of transaction 0","Could not open file ""pg_distributedlog/xxxx"": Operation not permitted.",

This could be related with security software running at segment host. Normally greenplum will not generate this alert "operation not permitted".

Environment

gpdb 6.x

Cause

From segment host, we can use "top" tool to check the process used the most CPU. One example could be "cbram" process.  cbram is a process from cybereason  (a security software).

Resolution

We can suggest stop this security software from each segment host first, then monitor for a while.