Scheduled file-based backups for vCenter to an SFTP target sporadically fail with "BackupManager encountered an exception. See logs for details."
search cancel

Scheduled file-based backups for vCenter to an SFTP target sporadically fail with "BackupManager encountered an exception. See logs for details."

book

Article ID: 386500

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Scheduled file-based backups for vCenter sporadically fail with an error message: 

BackupManager encountered an exception. See logs for details. https://<vcenter_fqdn>/appliance/support-bundle

Manually triggered backups using the same target/configuration however are working fine, as do most of the scheduled backups.

When looking at the /var/log/vmware/applmgmt/backup.log, you see the following error stack, reporting a "Broken pipe" during the update manager (VUM) component backup:

backup.log
..
YYYY-MM-DDThh:mm:ss:uuu [20250112-235904-24322831] [ComponentScriptsBackup:PID-3273501] [ComponentScripts::ComponentScriptsBackup:ComponentScripts.py:88] INFO: Execute vum script: /etc/vmware/backup/component-scripts/vum/backup_restore.py --startBackup
YYYY-MM-DDThh:mm:ss:uuu [20250112-235904-24322831] [ComponentScriptsBackup:PID-3273501] [ComponentScripts::ComponentScriptsBackup:ComponentScripts.py:95] INFO: Dispatching stream.
YYYY-MM-DDThh:mm:ss:uuu [20250112-235904-24322831] [ComponentScriptsBackup:PID-3273501] [Proc::GetProcsStatus:Proc.py:344] ERROR: Process returncode is -13, but expected exit codes are [0].
YYYY-MM-DDThh:mm:ss:uuu [20250112-235904-24322831] [ComponentScriptsBackup:PID-3273501] [Proc::GetProcsStatus:Proc.py:328] ERROR: rc: 1, stderr: Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/../util/Calculate.py", line 58, in <module>
    main(sys.argv[1], sys.argv[2], sys.argv[3])
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/../util/Calculate.py", line 45, in main
    stdout_obj.write(data)
BrokenPipeError: [Errno 32] Broken pipe
..
YYYY-MM-DDThh:mm:ss:uuu [20250112-235904-24322831] [ComponentScriptsBackup:PID-3273501] [Proc::GetProcsStatus:Proc.py:333] INFO: Skip to report the error.
YYYY-MM-DDThh:mm:ss:uuu [20250112-235904-24322831] [ComponentScriptsBackup:PID-3273501] [Proc::GetProcsStatus:Proc.py:344] ERROR: Process returncode is 1, but expected exit codes are [0].
YYYY-MM-DDThh:mm:ss:uuu [20250112-235904-24322831] [ComponentScriptsBackup:PID-3273501] [Proc::UpdateExceptionStatus:Proc.py:384] ERROR: Checksum not generated at /dev/shm/backupRestoreSumFile-<ID>
YYYY-MM-DDThh:mm:ss:uuu [20250112-235904-24322831] [ComponentScriptsBackup:PID-3273501] [ComponentScripts::ComponentScriptsBackup:ComponentScripts.py:106] ERROR: Component backup command "/etc/vmware/backup/component-scripts/vum/backup_restore.py --startBackup" failed 1.
YYYY-MM-DDThh:mm:ss:uuu [20250112-235904-24322831] [ComponentScriptsBackup:PID-3273501] [Log::run:Log.py:64] ERROR: Unable to perform backup: [Errno 32] Broken pipe
YYYY-MM-DDThh:mm:ss:uuu [20250112-235904-24322831] [ComponentScriptsBackup:PID-3273501] [ComponentScripts::ComponentScriptsBackup:ComponentScripts.py:135] ERROR: Error during component vum backup
Underlying process status. rc: 1
stdout:
stderr:
Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/ComponentScripts.py", line 110, in ComponentScriptsBackup
    raise BackupRestoreError(('Error during component %s backup' %
util.Common.BackupRestoreError: Error during component vum backup
Underlying process status. rc: 1
stdout:
stderr:
YYYY-MM-DDThh:mm:ss:uuu [20250112-235904-24322831] [MainProcess:PID-3273234] [Proc::VerifyProcStatusAndGetArchive:Proc.py:159] ERROR: Error at process ComponentScriptsBackup; rc:1.
YYYY-MM-DDThh:mm:ss:uuu [20250112-235904-24322831] [MainProcess:PID-3273234] [Proc::VerifyProcStatusAndGetArchive:Proc.py:163] ERROR: stderr:Error during component vum backup

YYYY-MM-DDThh:mm:ss:uuu[20250112-235904-24322831] [MainProcess:PID-3273234] [Proc::VerifyProcStatusAndGetArchive:Proc.py:172] INFO: Following error message isn't localized:
  stderr:Error during component vum backup

 

When looking at the component backup log for for update manager itself, /var/log/vmware/vmware-updatemgr/backup-restore.log, you find the same broken pipe error:

vmware-updatemgr/backup-restore.log
..
YYYY-MM-DDThh:mm:ss ERROR [3273560] [MainThread] [backup_restore:82] Failed to complete backup.
Traceback (most recent call last):
  File "/etc/vmware/backup/component-scripts/vum/backup_restore.py", line 79, in backup
    tar.add(PATCH_STORE_DIR, filter=skipFile)
  File "/usr/lib/python3.10/tarfile.py", line 2179, in add
    self.add(os.path.join(name, f), os.path.join(arcname, f),
  File "/usr/lib/python3.10/tarfile.py", line 2179, in add
    self.add(os.path.join(name, f), os.path.join(arcname, f),
  File "/usr/lib/python3.10/tarfile.py", line 2179, in add
    self.add(os.path.join(name, f), os.path.join(arcname, f),
  [Previous line repeated 2 more times]
  File "/usr/lib/python3.10/tarfile.py", line 2173, in add
    self.addfile(tarinfo, f)
  File "/usr/lib/python3.10/tarfile.py", line 2201, in addfile
    copyfileobj(fileobj, self.fileobj, tarinfo.size, bufsize=bufsize)
  File "/usr/lib/python3.10/tarfile.py", line 255, in copyfileobj
    dst.write(buf)
  File "/usr/lib/python3.10/tarfile.py", line 441, in write
    self.__write(s)
  File "/usr/lib/python3.10/tarfile.py", line 449, in __write
    self.fileobj.write(self.buf[:self.bufsize])
BrokenPipeError: [Errno 32] Broken pipe

Environment

vCenter Server 7.x

vCenter Server 8.x

Cause

This kind of issue is usually caused by the backup target or the connection between vCenter and target being overloaded by competing backups, usually because of tightly scheduled backup windows for the environment.

 

Resolution

This is not a failure of the vCenter Server itself, but usually caused by resource contention on the backup target or in the connection between vCenter and backup target.

To prevent these sporadic backup issues, try to relax the amount of competing operations, by moving the scheduled vCenter backup to a different time of day,

Powered by Wolken Software