vCenter Server fails to start with "Remote login failed:N3Vim5Fault9HttpFault9Exception", after it's restored from backup or snapshot, or a failed upgrade

Products

VMware vCenter Server

Issue/Introduction

After performing a restore of vCenter Server Appliance (VCSA) from a backup or rolling back to a previous snapshot, or a failed major version upgrade (Ex. 8.0 to 9.1), the vCenter services fails to start.

In /var/log/vmware/vpxd/vpxd.log entries similar to below are seen:

YYYY-MM-DDThh:mm:ss.msZ warning vpxd[7F38BD9C2700] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f38a41cc910, h:32, <TCP '0.0.0.0:0'>, <TCP '127.0.0.#:#####'>>, e: system:111(Connection refused)
YYYY-MM-DDThh:mm:ss.msZ warning vpxd[7F38CC164800] [Originator@6876 sub=VpxdAuthClient] [ConnectAndLogin] Failed to loginBySamlToken: N7Vmacore15SystemExceptionE(Connection refused)
--> [context]#####################################################################[/context]
YYYY-MM-DDThh:mm:ss.msZ info vpxd[7F38CC164800] [Originator@6876 sub=VpxdAuthClient] fallback to loginByCertificate
YYYY-MM-DDThh:mm:ss.msZ warning vpxd[7F38B7C78700] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f38a41c4310, h:32, <TCP '0.0.0.0:0'>, <TCP '127.0.0.#:#####>>, e: system:111(Connection refused)
YYYY-MM-DDThh:mm:ss.msZ error vpxd[7F38CC164800] [Originator@6876 sub=httpUtil] [HttpUtil::ExecuteRequest] Error in sending request - Connection refused
YYYY-MM-DDThh:mm:ss.msZ error vpxd[7F38CC164800] [Originator@6876 sub=ServerAccess] Remote login failed: N3Vim5Fault9HttpFault9ExceptionE(vim.fault.HttpFault)
--> [context]####################################################################[/context]
YYYY-MM-DDThh:mm:ss.msZ error vpxd[7F38CC164800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Failed to connect to IS: <N5Vmomi5Fault17HostCommunication9ExceptionE(vmodl.fault.HostCommunication)</font>
--> [context]#####################################################################[/context]>

YYYY-MM-DDThh:mm:ss.msZ error vpxd[7F38CC164800] [Originator@6876 sub=Authorize] Failed to initialize authorizeManager
YYYY-MM-DDThh:mm:ss.msZ warning vpxd[7F38CC164800] [Originator@6876 sub=VpxProfiler] Start [VpxdAuthorize::Start()] took 183263 ms
YYYY-MM-DDThh:mm:ss.msZ warning vpxd[7F38CC164800] [Originator@6876 sub=VpxProfiler] ServerApp::Start [TotalTime] took 183356 ms
YYYY-MM-DDThh:mm:ss.msZ error vpxd[7F38CC164800] [Originator@6876 sub=Default] Failed to start VMware VirtualCenter. Shutting down
YYYY-MM-DDThh:mm:ss.msZ info vpxd[7F38CC164800] [Originator@6876 sub=SupportMgr] Wrote uptime information
YYYY-MM-DDThh:mm:ss.msZ info vpxd[7F38BE15F700] [Originator@6876 sub=HostGateway] CmConnectionFSM::RunFSM(ST_CM_CALL_FAILED)
YYYY-MM-DDThh:mm:ss.msZ warning vpxd[7F38BE15F700] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Invalid argument : cmStub
YYYY-MM-DDThh:mm:ss.msZ warning vpxd[7F38BE15F700] [Originator@6876 sub=HostGateway] State(ST_ACQUIRE_TOKEN) failed with: Invalid argument : stsClient
YYYY-MM-DDThh:mm:ss.msZ info vpxd[7F38BE15F700] [Originator@6876 sub=HostGateway] stsUrlFromConfig: https://example-psc.example.com/sts/STSService/vsphere.local ssoAdminUrlFromConfig: https://example-psc.example.com/sso-adminserver/sdk/vsphere.local
YYYY-MM-DDThh:mm:ss.msZ info vpxd[7F38BE15F700] [Originator@6876 sub=vpxCrypt] Failed to read X509 cert; err: 151441516
YYYY-MM-DDThh:mm:ss.msZ info vpxd[7F38BE15F700] [Originator@6876 sub=vpxCrypt] Failed to read X509 cert; err: 151441516

In /var/log/vmware/vmafdd/vmafdd-syslog.log entries similar to below are seen:

YYYY-MM-DDThh:mm:ss.ms.213900+00:00 err vmafdd t@139929541560064: VmAfSrvCfgGetMachineID failed. Error(9234)
YYYY-MM-DDThh:mm:ss.ms.214035+00:00 err vmafdd t@139929541560064: [Error - 9234, ../../../server/vmafd/config.c:440]
YYYY-MM-DDThh:mm:ss.ms.214166+00:00 err vmafdd t@139929541560064: VmAfSrvGetCMLocation failed. Error(9234)
YYYY-MM-DDThh:mm:ss.ms.214295+00:00 err vmafdd t@139929541560064: ERROR! [VmAfdIpcGetCMLocation] is returning [9234]
YYYY-MM-DDThh:mm:ss.ms.020892+00:00 err vmafdd t@139929716352768: [Error - 9234, ../../../server/vmafd/ldap.c:170]
YYYY-MM-DDThh:mm:ss.ms.021971+00:00 err vmafdd t@139929716352768: [Error - 9234, ../../../server/vmafd/rootfetch.c:256]
YYYY-MM-DDThh:mm:ss.ms.022297+00:00 notice vmafdd t@139929716352768: Failed to update trusted roots. Error [9234]
YYYY-MM-DDThh:mm:ss.ms.026505+00:00 err vmafdd t@139929564739328: [Error - 9234, ../../../server/vmafd/ldap.c:170]
YYYY-MM-DDThh:mm:ss.ms.704271+00:00 err vmafdd t@139929541560064: [Error - 9234, ../../../server/vmafd/vmdir.c:1468]
YYYY-MM-DDThh:mm:ss.ms.704528+00:00 notice vmafdd t@139929541560064: VmAfSrvDirOpenConnection failed. Error(9234)
YYYY-MM-DDThh:mm:ss.ms.704686+00:00 err vmafdd t@139929541560064: [Error - 9234, ../../../server/vmafd/vmdir.c:1023]
YYYY-MM-DDThh:mm:ss.ms.704825+00:00 notice vmafdd t@139929541560064: VmAfSrvDirGetMachineId failed. Error(9234)
YYYY-MM-DDThh:mm:ss.ms.704983+00:00 err vmafdd t@139929541560064: [Error - 9234, ../../../server/vmafd/config.c:1307]

When attempting to execute one of the VMAFD-CLI get-* or set-* commands, we see output similar to below:

/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost


Error 9234: Authentication to VMware Directory Service failed.
Verify the username and password.
 
/usr/lib/vmware-vmafd/bin/vmafd-cli set-dc-name --server-name localhost --dc-name example-vc.example.com
Error 9234: Authentication to VMware Directory Service failed.
Verify the username and password.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server

Cause

Periodically, the password used by the Machine Account is automatically refreshed.
This password is stored on the Machine Account on vCenter Server and within the VMware Directory Service residing on the Platform Services Controllers. If the backup used to perform a restore, which pre-dates the most recent password refresh, upon restore the Machine Account will be unable to log into the VMware Directory Service.
Additionally, if the vCenter Server is rolled back to a snapshot state which pre-dates the most recent password refresh, upon restore the Machine Account will be unable to log into the VMware Directory Service.
This can also occur during a failed major version upgrade when reverting back to the original vCenter appliance.

Resolution

To update the password used by the vCenter Server's Machine Account after a restore, use the vcenter-restore CLI to issue a reset request. For more information, see the Restore the Failed vCenter Server Virtual Machine With the Direct-to-Host Emergency Restore Operation section in the vCenter Server Installation and Setup Guide.

For vCenter Server Appliance

Login to vCenter Server via SSH as the root user
Run this command to switch to BASH
```
shell
```
Run this command to Stop all of the vCenter Server services:
```
service-control --stop --all
```
Run this command Below does not work on embedded node:
```
vcenter-restore -u administrator -p <[email protected] password>

For example:
vcenter-restore -u administrator -p PASSWORD
```
Note: This CLI can be executed without either option, and the user will be prompted to provide the administrator and [email protected] password.
Run this command to confirm that all services are now started:
```
service-control --status --all
```

To manually invoke an update vCenter Server Machine Account password, use the dir-cli computer password-reset CLI to issue a reset request.

For vCenter Server Appliance

Login to the vCenter Server via SSH as the root user
Run this command to switch to the BASH shell:
```
shell
```

Run the command below:

/usr/lib/vmware-vmafd/bin/dir-cli computer password-reset --login administrator --live-dc-hostname <Platform Services Controller FQDN> --password <[email protected] password>

For example:

/usr/lib/vmware-vmafd/bin/dir-cli computer password-reset --login administrator --live-dc-hostname example-psc.example.com --password 'PASSWORD'

A successful password reset will output:

Password for machine account reset.

Run below command to Stop and Start the vCenter Server services:
```
service-control --stop --all && service-control --start --all
```