vCenter Restore fail during reconciliation.

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

The SSO credential validation was removed before starting stage II restore. So, even if the credentials are invalid, the stage II restore starts but fails during reconciliation without a proper error on the UI. This article is to debug such restore failures.

Symptoms:
VC restore fails while resetting machine account password during reconciliation in stage II.

Reconciliation logs(/var/log/vmware/applmgmt/reconciliation.log) will have error message similar to below snippet.

2021-10-19T06:33:56.636 [MainProcess:PID-5477] INFO: Resetting machine account password in lotus..
2021-10-19T06:33:56.719 [MainProcess:PID-5477] ERROR: Command exited with non-zero return code 246. Stdout   Stderr dir-cli failed. Error 382312694: Access denied, reason = rpc_s_auth_method (0x16c9a0f6).

2021-10-19T06:33:56.719 [MainProcess:PID-5477] ERROR: Failed to complete reconciliation.
Error Failed to reset machine account password.
2021-10-19T06:33:57.398 [MainProcess:PID-5477] ERROR: Failed to post event com.vmware.applmgmt.reconcil.job.failed.event to http://localhost:8085/sdk. Err: [Errno 111] Connection refused

Environment

VMware vSphere 6.7.x

Cause

The SSO credentials mentioned to start the stage II restore are incorrect. But, since validation of these credentials is removed, this issue can be seen when restore is started with incorrect SSO credentials.

Resolution

Currently there is no resolution.

Workaround:
Restart the restore with correct credentials.