VCSA VAMI Backup Fails During vmdird Database Backup: "BackupManager encountered an exception"

Products

• VMware vCenter Server

Issue/Introduction

• Attempts to perform a file-based backup using the vCenter Server VAMI UI fail with the following error message:

BackupManager encountered an exception

• vCenter Server Appliance (VCSA) backup via vCenter Appliance Management Interface (VAMI) fails during lotus database backup.
  
  
• vCenter server appliance (VCSA ) backup  via vCenter Appliance Management Interface (VAMI) fails at 5%.
  
  
• VMware Directory Service (vmdird) status check using vdcadmintool or dir-cli returns as Standalone (8).

/usr/lib/vmware-vmafd/bin/dir-cli state get
Enter password for [email protected]:

Directory Server State: Standalone (8)

• /var/log/vmware/applmgmt/backup.log file will contain one of the following error snippets:

yyyy-mm-ddThh:mm:ss [LotusBackup:PID-######] [Lotus::BackupLotus:Lotus.py:##] INFO: Starting backup Lotus.
yyyy-mm-ddThh:mm:ss [LotusBackup:PID-######] [Lotus::BackupLotus:Lotus.py:##] INFO: BackupLotus: Running /usr/lib/vmware-vmdir/bin/vdcbackup /storage/db/vmware-vmdir/ /tmp/backup_lotus
yyyy-mm-ddThh:mm:ss [LotusBackup:PID-######] [Lotus::BackupLotus:Lotus.py:##] INFO: stdout: VdcBackupDB: Server in state (8) in which it cannot perform backup. error(9125)
yyyy-mm-ddThh:mm:ss [LotusBackup:PID-######] [Lotus::BackupLotus:Lotus.py:##] ERROR: BackupLotus: command "/usr/lib/vmware-vmdir/bin/vdcbackup /storage/db/vmware-vmdir/ /tmp/backup_lotus" failed. rc: 165.
yyyy-mm-ddThh:mm:ss [LotusBackup:PID-######] [Lotus::BackupLotusCleanup:Lotus.py:###] INFO: Successfully completed Lotus cleanup.
yyyy-mm-ddThh:mm:ss [MainProcess:PID-######] [vpxdevent_lib::dispatch_events:vpxdevent_lib.py:###] INFO: Event com.vmware.applmgmt.backup.job.failed.event successfully posted tohttp://localhost:8085/sdk
yyyy-mm-ddThh:mm:ss [MainProcess:PID-######] [BackupManager::main:BackupManager.py:###] INFO: Backup job failed.

OR

yyyy-mm-ddThh:mm:ss [MainProcess:PID-######] [Proc::VerifyProcStatusAndGetArchive:Proc.py:###] ERROR: Error at process LotusBackup; rc:165.
yyyy-mm-ddThh:mm:ss [MainProcess:PID-######] [Proc::VerifyProcStatusAndGetArchive:Proc.py:###] ERROR: stderr:Encounter error during dump Lotus, check log for details: backup.log
yyyy-mm-ddThh:mm:ss [MainProcess:PID-######] [Proc::VerifyProcStatusAndGetArchive:Proc.py:###] INFO: Following error message isn't localized:  stderr:Encounter error during dump Lotus, check log for details: backup.log
yyyy-mm-ddThh:mm:ss [MainProcess:PID-######] [BackupManager::main:BackupManager.py:###] ERROR: BackupManager encountered an exception: Hit exception inside process LotusBackup:
yyyy-mm-ddThh:mm:ss [MainProcess:PID-######] [BackupManager::HandleBackupCleanup:BackupManager.py:###] INFO: Cleaning up the backup job.

Environment

VMware vCenter Server 7.x

VMware vCenter Server 8.x

Cause

This issue occurs because the VMware Directory Service (vmdird) on the vCenter Server Appliance (VCSA) is currently operating in 'standalone' mode.

Since a standalone state breaks directory replication across the Enhanced Linked Mode (ELM) topology, authentication and service lookup processes fail. Restoring the node to a replicated state is required to resolve the synchronization issues.

Resolution

1. Take snapshot of the vCenter Server Appliance, for more information check Snapshot Best practices for vCenter Server Appliance.

2. Log in to the VCSA via ssh and run the shell command.

3. Validate the entry in the likewise registry:

/opt/likewise/bin/lwregshell list_values '[HKEY_THIS_MACHINE\Services\vmdir]' | grep "Arguments"

+  "Arguments"         REG_SZ          "/usr/lib/vmware-vmdir/sbin/vmdird -s -l 0 -f /usr/lib/vmware-vmdir/share/config/vmdirschema.ldif -m standalone"

 

Note: If the argument value does not have -m standalone appended, execute command /usr/lib/vmware-vmafd/bin/dir-cli state set --state NORMAL to update the vmdird state to NORMAL and reboot the vCenter Server Appliance (VCSA).

 

4. Modify the likewise registry to update the value.

• For vCenter 8.0 U3.

/opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\vmdir]' "Arguments" "/usr/lib/vmware-vmdir/sbin/vmdird -L /var/log/vmware/vmdird/vmdird.log -l 0 -f /usr/lib/vmware-vmdir/share/config/vmdirschema.ldif"

• For versions prior to 8.0 U3.

/opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\vmdir]' "Arguments" "/usr/lib/vmware-vmdir/sbin/vmdird -s -l 0 -f /usr/lib/vmware-vmdir/share/config/vmdirschema.ldif"

 

5. Reboot the VCSA for the change to take effect.

Additional Information

Japanese version : VAMI を介した VCSA バックアップが、vmdird データベースのバックアップ中に「Server in state (8) in which it cannot perform backup」のエラーで失敗する