VCSA backup via VAMI fails during vmdird database backup with error "Server in state (8) in which it cannot perform backup"

Products

• VMware vCenter Server

Issue/Introduction

• vCenter Server Appliance (VCSA) backup via vCenter Appliance Management Interface (VAMI) fails during lotus database backup.
• vCenter server appliance (VCSA ) backup  via vCenter Appliance Management Interface (VAMI) fails at 5%.
• VMware Directory Service (vmdird) status check using vdcadmintool or dir-cli returns as Standalone (8).

/usr/lib/vmware-vmafd/bin/dir-cli state get
Enter password for [email protected]:

Directory Server State: Standalone (8)

• /var/log/vmware/applmgmt/backup.log file will contain one of the following error snippets:

yyyy-mm-ddThh:mm:ss [LotusBackup:PID-######] [Lotus::BackupLotus:Lotus.py:##] INFO: Starting backup Lotus.
yyyy-mm-ddThh:mm:ss [LotusBackup:PID-######] [Lotus::BackupLotus:Lotus.py:##] INFO: BackupLotus: Running /usr/lib/vmware-vmdir/bin/vdcbackup /storage/db/vmware-vmdir/ /tmp/backup_lotus
yyyy-mm-ddThh:mm:ss [LotusBackup:PID-######] [Lotus::BackupLotus:Lotus.py:##] INFO: stdout: VdcBackupDB: Server in state (8) in which it cannot perform backup. error(9125)
yyyy-mm-ddThh:mm:ss [LotusBackup:PID-######] [Lotus::BackupLotus:Lotus.py:##] ERROR: BackupLotus: command "/usr/lib/vmware-vmdir/bin/vdcbackup /storage/db/vmware-vmdir/ /tmp/backup_lotus" failed. rc: 165.
yyyy-mm-ddThh:mm:ss [LotusBackup:PID-######] [Lotus::BackupLotusCleanup:Lotus.py:###] INFO: Successfully completed Lotus cleanup.
yyyy-mm-ddThh:mm:ss [MainProcess:PID-######] [vpxdevent_lib::dispatch_events:vpxdevent_lib.py:###] INFO: Event com.vmware.applmgmt.backup.job.failed.event successfully posted tohttp://localhost:8085/sdk
yyyy-mm-ddThh:mm:ss [MainProcess:PID-######] [BackupManager::main:BackupManager.py:###] INFO: Backup job failed.

OR

yyyy-mm-ddThh:mm:ss [MainProcess:PID-######] [Proc::VerifyProcStatusAndGetArchive:Proc.py:###] ERROR: Error at process LotusBackup; rc:165.
yyyy-mm-ddThh:mm:ss [MainProcess:PID-######] [Proc::VerifyProcStatusAndGetArchive:Proc.py:###] ERROR: stderr:Encounter error during dump Lotus, check log for details: backup.log
yyyy-mm-ddThh:mm:ss [MainProcess:PID-######] [Proc::VerifyProcStatusAndGetArchive:Proc.py:###] INFO: Following error message isn't localized:  stderr:Encounter error during dump Lotus, check log for details: backup.log
yyyy-mm-ddThh:mm:ss [MainProcess:PID-######] [BackupManager::main:BackupManager.py:###] ERROR: BackupManager encountered an exception: Hit exception inside process LotusBackup:
yyyy-mm-ddThh:mm:ss [MainProcess:PID-######] [BackupManager::HandleBackupCleanup:BackupManager.py:###] INFO: Cleaning up the backup job.

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x

Cause

This issue is caused due to vCenter Server Appliance (VCSA)'s Directory Server State (vmdird state) set to standalone mode.

Resolution

1. Take snapshot of the vCenter Server Appliance, for more information check Snapshot Best practices for vCenter Server Appliance.

2. Log in to the VCSA via ssh and run the shell command.

3. Validate the entry in the likewise registry:

/opt/likewise/bin/lwregshell list_values '[HKEY_THIS_MACHINE\Services\vmdir]' | grep "Arguments"

+  "Arguments"         REG_SZ          "/usr/lib/vmware-vmdir/sbin/vmdird -s -l 0 -f /usr/lib/vmware-vmdir/share/config/vmdirschema.ldif -m standalone"

 

Note: If the argument value does not have -m standalone appended, execute command /usr/lib/vmware-vmafd/bin/dir-cli state set --state NORMAL to update the vmdird state to NORMAL and reboot the vCenter Server Appliance (VCSA).

 

4. Modify the likewise registry to update the value.

• For vCenter 8.0 U3.

/opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\vmdir]' "Arguments" "/usr/lib/vmware-vmdir/sbin/vmdird -L /var/log/vmware/vmdird/vmdird.log -l 0 -f /usr/lib/vmware-vmdir/share/config/vmdirschema.ldif"

• For versions prior to 8.0 U3.

/opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\vmdir]' "Arguments" "/usr/lib/vmware-vmdir/sbin/vmdird -s -l 0 -f /usr/lib/vmware-vmdir/share/config/vmdirschema.ldif"

 

5. Reboot the VCSA for the change to take effect.