gpbackup fails with ERROR 38000 "Device or resource busy" on Nutanix NFS

Products

• VMware Tanzu Data
• VMware Tanzu Greenplum
• VMware Tanzu Greenplum / Gemfire

Issue/Introduction

Intermittent gpbackup failures across various segments with the following error signature:

ERROR: child process exited with exit code 1 (segX x.x.x.x:600x pid=nnnnnn) (SQLSTATE 38000)command error message: 3rd party error log: could not redirect stdin to PROGRAM: Device or resource busy

The error typically occurs during the execution of COPY ... TO PROGRAM commands (e.g., while piping data to gzip).

Environment

Product: VMware Tanzu Greenplum
Versions Affected: gpbackup 1.x, Greenplum 6.x
Storage Layer: Nutanix Files (NFSv3)

Cause

This issue is typically environmental rather than a defect in the Greenplum software. Common root causes include:

1. Third-Party Locks: Antivirus, endpoint security, or backup agents (like Nutanix snapshots) may be locking the backup directory or the named pipes used by gpbackup during execution.
   
   
2. NFSv3 Locking Limitations: Under high concurrency (high --jobs parameter), the NFSv3 locking mechanism may encounter contention where a "Device or resource busy" state is returned if a lock cannot be acquired or released fast enough.

Resolution

1. Storage Optimization:
   • If possible, migrate to NFSv4.1, which offloads state management and locking to the server side, significantly reducing locking contention.
   • Verify I/O latency on the Nutanix Controller VMs (CVM) during the failure window to identify storage-level bottlenecks.
2. Exclusions: Ensure the Greenplum data directories and the /gpbackup mount point are strictly excluded from all antivirus and file-system monitoring tools.
   
   
3. Concurrency Control: Reduce the --jobs parameter in the gpbackup command to decrease the number of simultaneous write requests to the NFS mount.
   
   
4. Identify Locking Process: If the error occurs, run lsof <backup_path> on the affected segment host to identify the specific process holding the lock.