Cannot backup the vCenter due to backup.log file filling up
search cancel

Cannot backup the vCenter due to backup.log file filling up

book

Article ID: 318182

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 7.0

Issue/Introduction

  • When viewing the backup.log file on the vCenter server, the log file is full of messages such as the following in the /var/log/vmware/applmgmt/backup.log:

yyyy-mm-ddThh:mm:ss.mss  [VCDB-WAL-Backup:PID-52095] [VCDB::_backup_wal_files:VCDB.py:798] INFO: VCDB backup WAL start not received yet.
yyyy-mm-ddThh:mm:ss.mss [VCDB-WAL-Backup:PID-12251] [VCDB::_backup_wal_files:VCDB.py:798] INFO: VCDB backup WAL start not received yet.
yyyy-mm-ddThh:mm:ss.mss [VCDB-WAL-Backup:PID-29324] [VCDB::_backup_wal_files:VCDB.py:798] INFO: VCDB backup WAL start not received yet.
yyyy-mm-ddThh:mm:ss.mss [VCDB-WAL-Backup:PID-52095] [VCDB::_backup_wal_files:VCDB.py:798] INFO: VCDB backup WAL start not received yet.
yyyy-mm-ddThh:mm:ss.mss [VCDB-WAL-Backup:PID-12251] [VCDB::_backup_wal_files:VCDB.py:798] INFO: VCDB backup WAL start not received yet.
yyyy-mm-ddThh:mm:ss.mss [VCDB-WAL-Backup:PID-29324] [VCDB::_backup_wal_files:VCDB.py:798] INFO: VCDB backup WAL start not received yet.
yyyy-mm-ddThh:mm:ss.mss [VCDB-WAL-Backup:PID-52095] [VCDB::_backup_wal_files:VCDB.py:798] INFO: VCDB backup WAL start not received yet.
yyyy-mm-ddThh:mm:ss.mss [VCDB-WAL-Backup:PID-12251] [VCDB::_backup_wal_files:VCDB.py:798] INFO: VCDB backup WAL start not received yet.
yyyy-mm-ddThh:mm:ss.mss [VCDB-WAL-Backup:PID-29324] [VCDB::_backup_wal_files:VCDB.py:798] INFO: VCDB backup WAL start not received yet.
yyyy-mm-ddThh:mm:ss.mss [VCDB-WAL-Backup:PID-52095] [VCDB::_backup_wal_files:VCDB.py:798] INFO: VCDB backup WAL start not received yet.

  • Viewing the running processes shows numerous backup processes hung. Use the command on vCenter SSH: ps -auwwx | grep BackupManager.py

root     12194  0.0  0.2 130344 53776 ?        S    MMDD   hh:mm /usr/bin/python /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py
root     29266  0.0  0.1 130352 22184 ?        S    MMDD   hh:mm /usr/bin/python /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py
root     51966  0.0  0.1 130360 20160 ?        S    MMDD   hh:mm /usr/bin/python /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py
root     12235  0.0  0.0 141588 18924 ?        S    MMDD   hh:mm /usr/bin/python /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py
root     12251  0.0  0.0  67756 15352 ?        S    MMDD  hh:mm /usr/bin/python /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py
root     29306  0.0  0.0 141588 10004 ?        S    MMDD  hh:mm /usr/bin/python /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py
root     29324  0.0  0.0  67756  8392 ?        S    MMDD  hh:mm /usr/bin/python /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py
root     52076  0.0  0.0 141588  8176 ?        S    MMDD   hh:mm /usr/bin/python /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py
root     52095  0.0  0.0  67756  6744 ?        S    MMDD  hh:mm /usr/bin/python /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/BackupManager.py

Environment

VMware vCenter Server 7.0.3

Cause

This is caused by failed backups failing to clean up the VCDB process, which keeps waiting for the process to restart. 

Any subsequent attempts to back up will continue to add more log spew.

Resolution

This issue is resolved in vCenter Server version 7.0 U3i.To download go to support.broadcom.com

To workaround this issue, choose one of the below methods:

Workaround 1: Clear Old Backup Jobs by Rebooting

  1. Take a snapshot of the vCenter Server appliance. This snapshot can be used to revert to a previous state if necessary.
  2. Reboot the vCenter Server appliance. This will clear all old backup jobs.

Workaround 2: Clear Old Backup Jobs by Terminating Processes

  1. Take a snapshot of the vCenter Server appliance.
  2. Access the vCenter Server appliance via SSH.
  3. Identify the Process ID (PID) of the BackupManager.py process:
    • Run the following command: ps -auwwx | grep BackupManager.py

      This command will list processes running on the system. Look for entries containing "BackupManager.py". Note the PID (Process ID) associated with these entries. You may find multiple PIDs as BackupManager.py might have multiple threads running.
  4. Terminate the existing backup jobs:
    • Use the kill command to terminate the processes with the PIDs identified in step 3.
    • Important: Be cautious when using the kill command. Ensure you have the correct PIDs before proceeding kill -9 <PID from step 3>
    • Replace <PID from step 3> with the actual PID you obtained earlier. You may need to repeat this command for each identified PID.

After applying either workaround:

  • Verify that the old backup jobs are no longer running.
  • Consider scheduling regular backups to ensure data protection for your vCenter Server.
 
  •  

Additional Information

Impact/Risks:
Unable to troubleshoot a backup issue because the log messages are overrun with the above-mentioned log entries.

Powered by Wolken Software