vCenter Server Appliance File-Based Backup fails with "Broken pipe" and "No space left on device" errors

Products

• VMware vCenter Server

Issue/Introduction

When attempting to run scheduled or manual file-based backups in vCenter Server Appliance (vCSA) 8.0.3, the backup fails.

In the /var/log/vmware/applmgmt/applmgmt.log or /var/log/vmware/applmgmt/backup.log files, you see errors similar to the following:

[VCDBBackup:PID-2896696] [Proc::GetProcsStatus:Proc.py:328] ERROR: rc: 2, stderr: /usr/bin/tar: -: Cannot write: Broken pipe
/usr/bin/tar: Error is not recoverable: exiting now

[VCDBBackup:PID-2896696] [Proc::GetProcsStatus:Proc.py:328] ERROR: rc: 1, stderr: Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/../util/Calculate.py", line 45, in main
    stdout_obj.write(data)
BrokenPipeError: [Errno 32] Broken pipe

[VCDBBackup:PID-2896696] [Proc::GetProcsStatus:Proc.py:361] ERROR: rc: 1, stderr: b'cat: write error: No space left on device\n'

[MainProcess:PID-2896440] [Proc::VerifyProcStatusAndGetArchive:Proc.py:159] ERROR: Error at process ComponentScriptsBackup; rc:120.
[MainProcess:PID-2896440] [Proc::VerifyProcStatusAndGetArchive:Proc.py:163] ERROR: stderr:Error during component vum backup

Environment

VMware vCenter Server Appliance 8.0

Cause

This issue occurs when the destination backup server (e.g., SFTP, NFS, SMB) has reached its storage capacity or the specific backup user has reached its storage quota limit.

During the transfer of backup components (such as VCDB or VUM), the vCenter Server attempts to write data to the network stream. The destination server abruptly closes the connection ("Broken Pipe") because the destination volume is full and cannot accept additional data ("No space left on device").

Resolution

To resolve this issue, free up or expand the storage on the destination backup server.

1. Verify physical disk space on the target server: Ensure the destination partition has sufficient free space. It is recommended to have at least twice the size of the vCenter's used space available. (For Linux/SFTP servers, use df -h).

2. Verify User Quotas: Check if the specific user account used for authentication has hit a hard storage quota limit on the destination server.

3. Verify Inode usage: Check if the destination file system has run out of index nodes (inodes), which prevents new files from being written even if disk space is available. (For Linux/SFTP servers, use df -i).

Workaround:

To temporarily resume backups until the destination storage can be expanded, you can reduce the backup retention policy to a lower number or manually delete older, unneeded backup folders from the destination server to free up immediate space.

Additional Information

 

• For issues where "No space left on device" refers to local vCenter partitions (e.g., /storage/core or /storage/seat), refer to: vCenter Server VAMI backup fails with "Failed to dispatch full database files" (393817)

• For persistent "Broken pipe" errors when storage is available, refer to: VCSA after upgrade to version 8 is not able to complete backup (414580)