File-based backup in vCenter VAMI UI failing with stats and tasks option being selected

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• File-based backup using the vCenter VAMI UI fails with with an error message:

  BackupManager encountered an exception.

 

• The backup.log in /var/log/vmware/applmgmt contains the following errors:

  2021-11-16T13:08:55.648 [20211116-130837-18901211] [VCDBBackup:PID-19574] [VCDB::BackupVCDB:VCDB.py:2060] ERROR: Encounter error during backup VCDB.
  Traceback (most recent call last):
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/VCDB.py", line 1939, in BackupVCDB
      raise Exception('Full backup not allowed during VM snapshot')
  Exception: Full backup not allowed during VM snapshot
  2021-11-16T13:08:55.680 [20211116-130837-18901211] [VCDBBackup:PID-19574] [Proc::UpdateExceptionStatus:Proc.py:383] ERROR: Full backup not allowed during VM snapshot
  2021-11-16T13:08:55.681 [20211116-130837-18901211] [VCDBBackup:PID-19574] [telemetry::check_telemetry_enabled:telemetry.py:285] INFO: Check the telemetry conf file
  2021-11-16T13:08:55.681 [20211116-130837-18901211] [VCDBBackup:PID-19574] [telemetry::check_telemetry_enabled:telemetry.py:289] INFO: Check configuration parameter vcenter.telemetry
  2021-11-16T13:08:55.684 [20211116-130837-18901211] [VCDBBackup:PID-19574] [telemetry::check_telemetry_enabled:telemetry.py:293] INFO: Telemetry for Postgres is enabled
  2021-11-16T13:08:55.684 [20211116-130837-18901211] [VCDBBackup:PID-19574] [telemetry::get_vc_version:telemetry.py:301] INFO: Get vCenter version from /etc/vmware/.buildInfo
  2021-11-16T13:08:55.684 [20211116-130837-18901211] [VCDBBackup:PID-19574] [telemetry::get_vc_version:telemetry.py:307] INFO: vCenter version is 7.0U2b
  2021-11-16T13:08:55.685 [20211116-130837-18901211] [VCDBBackup:PID-19574] [VCDB::BackupVCDB:VCDB.py:2083] INFO: Telemetry command is ['/opt/vmware/vpostgres/current/bin/vmw_vpg_config/pg_trigger_telemetry.py', '--event-type', 'backup', '
  --data1', None, '--data3', '7.0U2b', '--data2', 'Success']
  2021-11-16T13:08:55.685 [20211116-130837-18901211] [VCDBBackup:PID-19574] [Proc::RunCmdForOutput:Proc.py:537] INFO: Executing command: ['/opt/vmware/vpostgres/current/bin/vmw_vpg_config/pg_trigger_telemetry.py', '--event-type', 'backup',
   '--data1', None, '--data3', '7.0U2b', '--data2', 'Success'].
  2021-11-16T13:08:55.689 [20211116-130837-18901211] [VCDBBackup:PID-19574] [VCDB::BackupVCDB:VCDB.py:2087] ERROR: Failed to push telemetry data for backup expected str, bytes or os.PathLike object, not NoneType
  Traceback (most recent call last):
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/VCDB.py", line 2084, in BackupVCDB
      RunCmdForOutput(logger, cmd)
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/util/Proc.py", line 541, in RunCmdForOutput
      close_fds=close_fds)
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/util/Proc.py", line 479, in ExecCmd
      shell=shell, cwd=cwd, close_fds=close_fds)
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/util/Proc.py", line 282, in CreateProcess
      shell=shell, cwd=cwd, close_fds=close_fds, env=current_env)
    File "/usr/lib/python3.7/subprocess.py", line 800, in __init__
      restore_signals, start_new_session)
    File "/usr/lib/python3.7/subprocess.py", line 1482, in _execute_child
      restore_signals, start_new_session, preexec_fn)
  TypeError: expected str, bytes or os.PathLike object, not NoneType

  2021-11-16T13:09:26.746 [20211116-130837-18901211] [MainProcess:PID-18868] [BackupManager::Cleanup:BackupManager.py:406] ERROR: Failed to clean up backup child processes.
  Traceback (most recent call last):
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py", line 650, in main
      backupObj.DoBackup()
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py", line 335, in DoBackup
      self.LaunchBackupProcesses()
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py", line 302, in LaunchBackupProcesses
      self.ExecBackupsInParallel()
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py", line 272, in ExecBackupsInParallel
      taskId=self.args.id, operation='BACKUP')
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/util/Proc.py", line 202, in LaunchMultipleProcesses
      timeout, logger)
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/util/Proc.py", line 178, in VerifyProcStatusAndGetArchive
      (procRecord.process.name, procRecord.status.excMsg))
  Exception: Hit exception inside process VCDBBackup: Full backup not allowed during VM snapshot

Environment

VMware vCenter Server Appliance 6.7.x

VMware vCenter Server 7.x

VMware vCenter Server 8.x 

Resolution

1.) Remove all snapshots from the vCenter VM.

2.) Remove file file backupMarker.txt by running the command:

• rm /etc/vmware/backupMarker.txt

3.) Re-attempt the backup with stats, events and task option selected.