vCenter Server restore fails at 91% with RestoreManager exception in Enhanced Linked Mode

Products

• VMware vCenter Server

Issue/Introduction

• Domain split after run the domain repoint command on existing vCenter.
• Existing vCenter crash and prompt restore after trying register already spilted vCenter.
• No offline snaphot taken before domain-repoint, restore from file based backup is required .
• vCenter restore via VAMI fails at 91% during the reconciliation phase, following error in /var/log/vmware/applmgmt/reconciliation.log

####-##-##T##:##:##.### [MainProcess:PID-6770] [svcsController::_svc_log:svcsController.py:71] INFO: Successfully started service vmafdd
####-##-##T##:##:##.### [MainProcess:PID-6770] [ReconciliationManager::_reconcile_replication_node:ReconciliationManager.py:124] INFO: Resetting machine account password and syncronizing data with partners.
####-##-##T##:##:##.### [MainProcess:PID-6770] [Common::ExecSecureCommand:Common.py:978] INFO: Spawning: /usr/bin/python  /usr/lib/vmware-vmdir/scripts/vmdirBackupRestore.py  --file  /storage/db/vmware-vmdir/partnerlist  --username  [email protected]  --reset  --restore  --autostart
####-##-##T##:##:##.### [MainProcess:PID-6770] [Common::ExecSecureCommand:Common.py:981] INFO: Supplying credential ...
####-##-##T##:##:##.### [MainProcess:PID-6770] [ReconciliationManager::_reconcile_replication_node:ReconciliationManager.py:135] ERROR: Command exited with non-zero return code 1. Stdout Validate partners complete^M
Error: Failed to reset machine account on partner DCs  Stderr Validate partners complete^M

Environment

VMware vCenter Server 8.x

Cause

• Misconfigured Domain Repoint: Join a new node to the existing ELM domain. However, domain repoint command mistakenly ran at existing node. This broke the replication partner status. 
• Accidental Unregistration: Unregistered crashed vc from working node. This action completely wiped the vc configuration from working node.
• Restore Failure: Because working node no longer had any record of crashed vc, the vc restore process continuously failed, as it required its replication partner to hold its historical data. See
  https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vcenter-installation-and-setup/file-based-backup-and-restore-of-a-vcenter-server-environment/restore-considerations-and-limitations.html

Resolution

1. Revert snapshot of the working node exists from before the incorrect unregistration.
2. Manually synchronize the [email protected] password across the all vCenter.
3. Restart the restore process for the failed vCenter node.

Additional Information

Ensure the virtual hardware (NICs) matches the vCenter backup configuration.