File-based backup in vCenter VAMI UI failing with stats and tasks option being selected

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• File-based backup using the vCenter VAMI UI fails with with an error message:

  BackupManager encountered an exception.

 

• the backup.log in /var/log/vmware/applmgmt contains the following errors:

  2021-11-16T13:08:55.648 [20211116-130837-18901211] [VCDBBackup:PID-19574] [VCDB::BackupVCDB:VCDB.py:2060] ERROR: Encounter error during backup VCDB.
  Traceback (most recent call last):
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/VCDB.py", line 1939, in BackupVCDB
      raise Exception('Full backup not allowed during VM snapshot')
  Exception: Full backup not allowed during VM snapshot
  2021-11-16T13:08:55.680 [20211116-130837-18901211] [VCDBBackup:PID-19574] [Proc::UpdateExceptionStatus:Proc.py:383] ERROR: Full backup not allowed during VM snapshot
  2021-11-16T13:08:55.681 [20211116-130837-18901211] [VCDBBackup:PID-19574] [telemetry::check_telemetry_enabled:telemetry.py:285] INFO: Check the telemetry conf file
  2021-11-16T13:08:55.681 [20211116-130837-18901211] [VCDBBackup:PID-19574] [telemetry::check_telemetry_enabled:telemetry.py:289] INFO: Check configuration parameter vcenter.telemetry
  2021-11-16T13:08:55.684 [20211116-130837-18901211] [VCDBBackup:PID-19574] [telemetry::check_telemetry_enabled:telemetry.py:293] INFO: Telemetry for Postgres is enabled
  2021-11-16T13:08:55.684 [20211116-130837-18901211] [VCDBBackup:PID-19574] [telemetry::get_vc_version:telemetry.py:301] INFO: Get vCenter version from /etc/vmware/.buildInfo
  2021-11-16T13:08:55.684 [20211116-130837-18901211] [VCDBBackup:PID-19574] [telemetry::get_vc_version:telemetry.py:307] INFO: vCenter version is 7.0U2b
  2021-11-16T13:08:55.685 [20211116-130837-18901211] [VCDBBackup:PID-19574] [VCDB::BackupVCDB:VCDB.py:2083] INFO: Telemetry command is ['/opt/vmware/vpostgres/current/bin/vmw_vpg_config/pg_trigger_telemetry.py', '--event-type', 'backup', '
  --data1', None, '--data3', '7.0U2b', '--data2', 'Success']
  2021-11-16T13:08:55.685 [20211116-130837-18901211] [VCDBBackup:PID-19574] [Proc::RunCmdForOutput:Proc.py:537] INFO: Executing command: ['/opt/vmware/vpostgres/current/bin/vmw_vpg_config/pg_trigger_telemetry.py', '--event-type', 'backup',
   '--data1', None, '--data3', '7.0U2b', '--data2', 'Success'].
  2021-11-16T13:08:55.689 [20211116-130837-18901211] [VCDBBackup:PID-19574] [VCDB::BackupVCDB:VCDB.py:2087] ERROR: Failed to push telemetry data for backup expected str, bytes or os.PathLike object, not NoneType
  Traceback (most recent call last):
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/VCDB.py", line 2084, in BackupVCDB
      RunCmdForOutput(logger, cmd)
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/util/Proc.py", line 541, in RunCmdForOutput
      close_fds=close_fds)
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/util/Proc.py", line 479, in ExecCmd
      shell=shell, cwd=cwd, close_fds=close_fds)
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/util/Proc.py", line 282, in CreateProcess
      shell=shell, cwd=cwd, close_fds=close_fds, env=current_env)
    File "/usr/lib/python3.7/subprocess.py", line 800, in __init__
      restore_signals, start_new_session)
    File "/usr/lib/python3.7/subprocess.py", line 1482, in _execute_child
      restore_signals, start_new_session, preexec_fn)
  TypeError: expected str, bytes or os.PathLike object, not NoneType

  2021-11-16T13:09:26.746 [20211116-130837-18901211] [MainProcess:PID-18868] [BackupManager::Cleanup:BackupManager.py:406] ERROR: Failed to clean up backup child processes.
  Traceback (most recent call last):
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py", line 650, in main
      backupObj.DoBackup()
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py", line 335, in DoBackup
      self.LaunchBackupProcesses()
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py", line 302, in LaunchBackupProcesses
      self.ExecBackupsInParallel()
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py", line 272, in ExecBackupsInParallel
      taskId=self.args.id, operation='BACKUP')
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/util/Proc.py", line 202, in LaunchMultipleProcesses
      timeout, logger)
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/util/Proc.py", line 178, in VerifyProcStatusAndGetArchive
      (procRecord.process.name, procRecord.status.excMsg))
  Exception: Hit exception inside process VCDBBackup: Full backup not allowed during VM snapshot

Environment

VMware vCenter Server Appliance 6.7.x
VMware vCenter Server 7.0.0

Resolution

• Remove all snapshots from the vCenter VM.
• Remove file file backupMarker.txt by running the command:

  # rm /etc/vmware/backupMarker.txt

• re-attempt the backup with stats, events and task option selected