gpcrondump fails with "OSError: [Errno 12] Cannot allocate memory"

Products

• VMware Tanzu Greenplum

Issue/Introduction

Symptoms:

When backing up a database/table etc., gpcrondump errors out at step "-Ensuring remaining free disk" with "Too many open files" or "Cannot allocate memory".

Example:

20150401:11:06:38:011302 gpcrondump:mdw:gpadmin-[INFO]:-Vacuum mode type                     = Off
20150401:11:06:38:011302 gpcrondump:mdw:gpadmin-[INFO]:-Ensuring remaining free disk         > 10
20150401:11:08:37:011302 gpcrondump:mdw:gpadmin-[CRITICAL]:-gpcrondump failed. (Reason='error 'ERROR:  could not create temporary file base/17227/pgsql_tmp/workfile_set_HashJoin_Slice-1.XXXXWeXZe6/spillfile_f129609:Too many open files
ERROR:  could not seek in temporary file: Bad file descriptor
' in '

SELECT ALLTABLES.oid, ALLTABLES.schemaname, ALLTABLES.tablename FROM 

[....]
[....]
File "/usr/local/greenplum-db/lib/python/gppylib/commands/base.py", line 421, in execute
    stdout=subprocess.PIPE, close_fds=True)
  File "/usr/local/greenplum-db/ext/python/lib/python2.6/subprocess.py", line 595, in __init__
    errread, errwrite)
  File "/usr/local/greenplum-db/ext/python/lib/python2.6/subprocess.py", line 1009, in _execute_child
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

 

Environment

Cause

This error is caused by the bug 24591. It is a consequence of a python issue where multiple processes allocate too much memory or over-allocate file descriptors.

Resolution

Workaround

Use the "-b" option along with the gpcrondump command. This will make it skip the disk space check functionality.

Fix

Upgrade to GPDB 4.3.5.0.