vCenter File-Level backup fails with error "Backup Manager encountered an exception. See logs for details"
search cancel

vCenter File-Level backup fails with error "Backup Manager encountered an exception. See logs for details"

book

Article ID: 411109

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • When attempting a scheduled or manual file-based backup via the vCenter Management Interface (VAMI), the process fails. Users encounter the following error message: "Backup Manager encountered an exception. See logs for details."
  • In /var/log/vmware/applmgmt/backup.log entries similar to below are seen: 
    • YYYY-MM-DDTHH:MM:SS[StellarDbBackup:PID-#######] [backupRestoreDispatch::dispatchFiles:backupRestoreDispatch.py:371] INFO: tarCmd = ['/usr/bin/tar', '-cz', '-C', '/tmp/backup_stellardb/', '--ignore-failed-read', '--warning', 'no-file-ignored', 'stellar.db']
      YYYY-MM-DDTHH:MM:SS[LotusBackup:PID-#######] [Lotus::BackupLotus:Lotus.py:74] INFO: stdout: VdcBackupDB: Server in state (5) in which it cannot perform backup. error(9125)
      YYYY-MM-DDTHH:MM:SS[LotusBackup:PID-#######] [Lotus::BackupLotus:Lotus.py:80] ERROR: BackupLotus: command "/usr/lib/vmware-vmdir/bin/vdcbackup /storage/db/vmware-vmdir/ /tmp/backup_lotus" failed. rc: 165.
  • Checking VMdir status through vCenter SSH we see output similar to below:  
    • /usr/lib/vmware-vmdir/bin/vdcadmintool
    • Press Option "6" to see VMdir Status. check whether it's set to  Normal/ Read-only/Standalone/Failure " If it's set to Read-only /Standalone refer to KB Article on Step1 on Resolution 

  • Checking Replication Status through vCenter SSH we see output similar to below: 

    • /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator
    • Sample results: 
      • Partner: #######vCenter 
Host available:   Yes
Status available: No

My last change number: 100537
Partner has seen my change number: 99947
Partner is 590 changes behind.
  • Note: Since Host available status shows "Yes" and Status Available as "No", last change values are different for both, Partner shows "590" instead of "0"changes behind, this shows Replication is not Intact 100% that needs to be fixed.

Environment

  • VMware vCenter Server 7.X
  • VMware vCenter Server 8.X

 

Cause

  • This issue typically occurs in environments with Enhanced Linked Mode (ELM). The backup fails because the VMware Directory Service (vmdir) is not in a healthy state. This is often caused by a break in replication between vCenter nodes, leaving one node in a "Read Only" or "Failure" state (State 5).                                                        

 

Resolution

To resolve this issue with scheduled and manual VAMI file-based backups, address the Enhanced Linked Mode and replication problems between the vCenter nodes.

Step 1: Reboot the vCenter

  • Reboot the vCenter server and verify if vmdir state is in normal state. 
    • Take SSH of the vCenter and login with root credentials.
    • Run the below command to check the vmdir state. 
      • /usr/lib/vmware-vmdir/bin/vdcadmintool (Then option 6 to get vmdir state)
    • If the output should be "Normal"
    • If not, please proceed with Step 2. 

Step 2: Rebuilding Replication between vCenter

Note: Before proceeding, you must take an offline snapshot of all vCenter Server nodes in the Linked Mode group. This ensures you have a recovery point if the replication rebuild fails.

  1. Download fix-psc-master script from KB Article:
  2. Transfer the script using any file transfer application to /tmp folder of vCenter
  3. SSH to vCenter with root and unzip the fix-psc-master with command below 
    • cd /tmp
    • unzip fix-psc-master
    • cd fix-psc-master
  4. Rebuild vmdir with the help of Healthy PSC. Run the following command on the broken unhealthy node pointing to the good healthy node as denoted by --healthy-psc-fqdn. 
    • ./fixpsc rebuild --healthy-psc-fqdn <FQDN>
  5. Restart all the vCenter Services. 
    • service-control --stop --all && service-control --start --all
  6. Check the vmdir status now, if it's set to "Normal"  run the command below and press option "6" 
    • /usr/lib/vmware-vmdir/bin/vdcadmintool
  7. Check the Replication is intact between both the vCenter ,Host available and Status available should be "Yes" last changes should have same value and Partner changes should be "0" as shown on example below: 
    • /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator
    • Sample Output:  
      • root@Test-vCenter [ ~ ]# /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator
password:
Partner: <vCenter FQDN>
Host available:   Yes
Status available: Yes

My last change number: 100537
Partner has seen my change number: 100537
Partner is 0 changes behind.

Since we see host available and status available as "Yes" last change number values are same for both, and Partner shows "0" changes behind, which shows replication is intact
  8. Re-run VAMI backup from vCenter Appliance 

Additional Information

Powered by Wolken Software