VCSA restore fails at 97%

Products

VMware Cloud Foundation VMware vCenter Server

Issue/Introduction

There is an error showing Reconciliation Job Failed at 97%
After the manual restart of services, The vstats service does not start.
vCenter - The Reconciliation Job Failed due to an error in upgrading VCF, Management Domain, when restoring the VCSA from backup the restore is at 97%, and then the process stops.
Trying the restore more than once will always get the same error.

yyyy-mm-ddThh:mm:ss.Z [#########-#####-##########] [MainProcess:PID-###] [ReconciliationManager::main:ReconciliationManager.py:###] ERROR: Failed to complete reconciliation. Error Failed to start all services.

Manually restarting the services works but the process on the appliance did not complete correctly and on the manage vcenter:5480 interface, it looks as if stage 2 had not been completed. https://VCSAFQDN:5480/configurev2/#/)
At the /var/log/vmware/applmgmt/reconciliation.log you see the below error message stating -

[ReconciliationManager::startAllServices:ReconciliationManager.py:226] ERROR: Failed to start all the vCenter services. Error: >Failed to start services in profile ALL. RC=4, stderr=Failed to start vstats services. Error: A system error occurred. Check logs

VAMI behavior similar to that shown in the KB Accessing the VAMI (https://VCSA:5480) stuck on "Getting Started"/"Setup vCenter" page due to stale restoreReconciliation-history.json

Environment

VMware vCenter Server

Cause

The VCSA restoration fails to complete due to Failed to complete reconciliation. which is caused by the failure to start all services in the VCSA specially when in ELM and the ldap 49 error is present in vmdird-syslog.log

Resolution

If the vstats service is not starting try fixing it by running the script from the article: LDAP Error Code 49 : Reset Machine Account Password of vCenter Server Appliance using Shell Script
Restart vCenter services and the vstats service should start correctly.
Remove the following 2 files:
- /storage/applmgmt/backup_restore/restoreReconciliation-history.json
- /storage/applmgmt/backup_restore/backup-history.json
Move the /var/vmware/applmgmt/backupRestore-history.json file to /tmp
Remove file file backupMarker.txt by running the command -

rm /etc/vmware/backupMarker.txt

Check the content of /var/log/firstboot/fbInstall.json and replace it with the following -

{

"progress" : 100,

"status" : "success"

}

we replace it from -

to -

Restart VCSA services with command:

service-control --stop --all && service-control --start --all