SDDC-Manager restore fails during RestoreSDDCSystemConfiguration

Products

VMware Cloud Foundation

Issue/Introduction

SDDC Manager restore task fails at a sub-task 'Restore SDDC Manager System Configuration'

The below error will be found in /var/log/vmware/vcf/sddc-support/vcf-sos.log

yyyy-mm-ddThh:mm:ss [ERROR sddcmanagerrestore.py::start_sddc_manager_ui_app::273::restore-aca4251b-1d8c-4091-ab82-6746f50dbe97] Traceback (most recent call last):
  File "backup/sddcmanagerrestore.py", line 268, in start_sddc_manager_ui_app
Exception: SDDC Manager UI App not ready even after 3 restart attempts.

Also, the below error will be found in /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log

yyyy-mm-ddThh:mm:ss ERROR [9240b985d7304e43, 3395f4d980094c26] [services/pscUtils.js, init-pscs, attemptPSCInit:71] Caught error from await primaryPscInit
yyyy-mm-ddThh:mm:ss WARN [9240b985d7304e43, 3395f4d980094c26] [services/pscUtils.js, init-pscs, attemptPSCInitWithRetry:115]
100.109: VError: PSC Initilization attempt "1" failed: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Unable to retrieve iDP Metadata: 500 - "\"Error executing remote command via SSH: WARNING: Your password has expired.\\nPassword change required but no TTY available.\""
    at Object.initializationPscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:100:5)
    at attemptPSCInitWithRetry (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:108:46)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

Environment

VMware Cloud Foundation 4.x

VMware Cloud Foundation 5.x

Cause

The SDDC-Manager-UI-App did not come up as it has a dependency on the vCenter and the root credentials of the vCenter are expired.

Resolution

Currently there is no resolution

Workaround:

Reset the vCenter's root account credentials to the previous one(the password that is used before taking SDDC-Manager backup), unlock the account if locked, and then restore SDDC-Manager.

Steps to unlock the root account
1. SSH to vCenter using [email protected] where vsphere.local is your default SSO Domain.
2. Enable shell by running
```
shell.set --enable true
```
3. Open shell connection by running
```
shell
```
4. Reset the 'root' account password by running the below command, then enter the new credentials.
```
sudo passwd root
```
5. Wait for a minute and then log in as 'root' by running "su - root" and then enter the newly set password(twice if asks).
6. Open shell running.
```
shell
```
7. Unlock the password by running
```
/usr/sbin/faillock
```
8. Exit the session by running.
```
exit
```
Steps to reset the vCenter's root account credentials to the previous (the password that is used before taking SDDC-Manager backup):
1. Log in to vCenter Server Management (https://<vCenter IP/FQDN>:5480) using root account credentials
2. Go to "Administration" and then change the password.
  - (Note: As you cannot use any of your previous five passwords, this step may need to run 5 times to set the previous one.)
Perform SDDC-Manager Restore operation.