vCenter Server VAMI backup with NFS protocol fails with error "Failed to dispatch full database files"
search cancel

vCenter Server VAMI backup with NFS protocol fails with error "Failed to dispatch full database files"

book

Article ID: 393817

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • The backup initiated through vCenter VAMI UI fails with error message "BackupManager encountered an exception".

  • The backup logs under the vCenter Server reports following error messages - 
    Log location: /var/log/vmware/applmgmt/backup.log 

YYYY-MM-DDTHH:MM:SS [YYYYMMDD-HHMMSS-vCenter_Server_Build_Number] [ComponentScriptsBackup:PID-214659] [Log::run:Log.py:64] ERROR: Unable to perform backup: [Errno 32] Broken pipe Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
YYYY-MM-DDTHH:MM:SS [YYYYMMDD-HHMMSS-vCenter_Server_Build_Number] [ComponentScriptsBackup:PID-214659] [Log::run:Log.py:64] ERROR: BrokenPipeError: [Errno 32] Broken pipe
YYYY-MM-DDTHH:MM:SS [YYYYMMDD-HHMMSS-vCenter_Server_Build_Number] [ComponentScriptsBackup:PID-214659] [ComponentScripts::ComponentScriptsBackup:ComponentScripts.py:106] ERROR: Component backup command "/etc/vmware/backup/component-scripts/vum/backup_restore.py --startBackup" failed 120.
YYYY-MM-DDTHH:MM:SS [YYYYMMDD-HHMMSS-vCenter_Server_Build_Number] [ComponentScriptsBackup:PID-214659] [ComponentScripts::ComponentScriptsBackup:ComponentScripts.py:135] ERROR: Error during component vum backup
Underlying process status. rc: 120
stdout:
stderr:
Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/ComponentScripts.py", line 110, in ComponentScriptsBackup
    raise BackupRestoreError(('Error during component %s backup' %
util.Common.BackupRestoreError: Error during component vum backup
Underlying process status. rc: 120
stdout:
stderr:
YYYY-MM-DDTHH:MM:SS [YYYYMMDD-HHMMSS-vCenter_Server_Build_Number] [VCDB-WAL-Backup:PID-214671] [Proc::RunCmdForOutput:Proc.py:538] INFO: Executing command: ['/opt/vmware/vpostgres/current/bin/psql', '-U', 'postgres', '-p', '5432', '-At', '-c', 'SELECT pg_walfile_name(pg_current_wal_lsn());'].
YYYY-MM-DDTHH:MM:SS [YYYYMMDD-HHMMSS-vCenter_Server_Build_Number] [VCDB-WAL-Backup:PID-214671] [VCDB::_backup_wal_files:VCDB.py:821] INFO: Current WAL file is: 00000003000004D9000000F0
YYYY-MM-DDTHH:MM:SS [YYYYMMDD-HHMMSS-vCenter_Server_Build_Number] [VCDB-WAL-Backup:PID-214671] [VCDB::_backup_wal_files:VCDB.py:825] INFO: No new WAL files since last backup
YYYY-MM-DDTHH:MM:SS [YYYYMMDD-HHMMSS-vCenter_Server_Build_Number] [VCDBBackup:PID-214660] [Proc::GetProcsStatus:Proc.py:328] ERROR: rc: 2, stderr: /usr/bin/tar: -: Wrote only 2048 of 10240 bytes
/usr/bin/tar: Error is not recoverable: exiting now

YYYY-MM-DDTHH:MM:SS [YYYYMMDD-HHMMSS-vCenter_Server_Build_Number] [VCDBBackup:PID-214660] [Proc::GetProcsStatus:Proc.py:333] INFO: Skip to report the error.
YYYY-MM-DDTHH:MM:SS [YYYYMMDD-HHMMSS-vCenter_Server_Build_Number] [VCDBBackup:PID-214660] [Proc::GetProcsStatus:Proc.py:344] ERROR: Process returncode is 2, but expected exit codes are [0, 1].
YYYY-MM-DDTHH:MM:SS [YYYYMMDD-HHMMSS-vCenter_Server_Build_Number] [VCDBBackup:PID-214660] [Proc::GetProcsStatus:Proc.py:328] ERROR: rc: 1, stderr: Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/../util/Calculate.py", line 58, in <module>
    main(sys.argv[1], sys.argv[2], sys.argv[3])
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/../util/Calculate.py", line 45, in main
    stdout_obj.write(data)
BrokenPipeError: [Errno 32] Broken pipe

YYYY-MM-DDTHH:MM:SS [YYYYMMDD-HHMMSS-vCenter_Server_Build_Number] [VCDBBackup:PID-214660] [Proc::GetProcsStatus:Proc.py:333] INFO: Skip to report the error.
YYYY-MM-DDTHH:MM:SS [YYYYMMDD-HHMMSS-vCenter_Server_Build_Number] [VCDBBackup:PID-214660] [Proc::GetProcsStatus:Proc.py:344] ERROR: Process returncode is 1, but expected exit codes are [0].
YYYY-MM-DDTHH:MM:SS [YYYYMMDD-HHMMSS-vCenter_Server_Build_Number] [VCDBBackup:PID-214660] [Proc::UpdateExceptionStatus:Proc.py:384] ERROR: Checksum not generated at /dev/shm/backupRestoreSumFile-YYYYMMDD-HHMMSS-vCenter_Server_Build_Number-lq0yjafu
YYYY-MM-DDTHH:MM:SS [YYYYMMDD-HHMMSS-vCenter_Server_Build_Number] [VCDBBackup:PID-214660] [VCDB::BackupVCDB:VCDB.py:2053] ERROR: Failed to backup VCDB.
Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/VCDB.py", line 1997, in BackupVCDB
    full_db_checksum = _full_database_backup(
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/VCDB.py", line 1257, in _full_database_backup
    raise BackupRestoreError('Failed to dispatch full database files.',
util.Common.BackupRestoreError: Failed to dispatch full database files.
Underlying process status. rc: 28
stdout:
stderr:

YYYY-MM-DDTHH:MM:SS [YYYYMMDD-HHMMSS-vCenter_Server_Build_Number] [MainProcess:PID-214430] [BackupManager::main:BackupManager.py:645] ERROR: BackupManager encountered an exception: Hit exception inside process ComponentScriptsBackup: Checksum not generated at /dev/shm/backupRestoreSumFile-YYYYMMDD-HHMMSS-vCenter_Server_Build_Number-v_kfd2ed
YYYY-MM-DDTHH:MM:SS [YYYYMMDD-HHMMSS-vCenter_Server_Build_Number] [MainProcess:PID-214430] [BackupManager::HandleBackupCleanup:BackupManager.py:486] INFO: Cleaning up the backup job.

 

Resolution

  • Ensure sufficient space is available on the NFS storage where the backups are stored.
  • If storage space is low, delete older backups from the storage system to free up space.
  • Once the cleanup is complete, retry the vCenter Server VAMI backup. This should allow the new backup to proceed successfully.
Powered by Wolken Software