vCenter Server fails to start with "Remote login failed:N3Vim5Fault9HttpFault9ExceptionE(vim.fault.HttpFault)", After vCenter Server is restored from backup or snapshot
search cancel

vCenter Server fails to start with "Remote login failed:N3Vim5Fault9HttpFault9ExceptionE(vim.fault.HttpFault)", After vCenter Server is restored from backup or snapshot

book

Article ID: 338638

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

After restoring vCenter Server or vCenter Server Appliance from backup or a snapshot, you experience these symptoms:
 

  • In the /var/log/vmware/vpxd/vpxd.log file, you see entries similar to:
YYYY-MM-DDThh:mm:ss.msZ warning vpxd[7F38BD9C2700] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f38a41cc910, h:32, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:10080'>>, e: system:111(Connection refused)
YYYY-MM-DDThh:mm:ss.msZ warning vpxd[7F38CC164800] [Originator@6876 sub=VpxdAuthClient] [ConnectAndLogin] Failed to loginBySamlToken: N7Vmacore15SystemExceptionE(Connection refused)
--> [context]zKq7AVECAAAAAMJySwAPdnB4ZAAAkJ4rbGlidm1hY29yZS5zbwAAhE8bAOqXGADgXysAWVcjAApfJgD+YCYAhIMmAICKJgCfISQAb+0jAFbwIwB79isBVHQAbGlicHRocmVhZC5zby4wAAItjg5saWJjLnNvLjYA[/context]
YYYY-MM-DDThh:mm:ss.msZ info vpxd[7F38CC164800] [Originator@6876 sub=VpxdAuthClient] fallback to loginByCertificate
YYYY-MM-DDThh:mm:ss.msZ warning vpxd[7F38B7C78700] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f38a41c4310, h:32, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:10080'>>, e: system:111(Connection refused)
YYYY-MM-DDThh:mm:ss.msZ error vpxd[7F38CC164800] [Originator@6876 sub=httpUtil] [HttpUtil::ExecuteRequest] Error in sending request - Connection refused
YYYY-MM-DDThh:mm:ss.msZ error vpxd[7F38CC164800] [Originator@6876 sub=ServerAccess] Remote login failed: N3Vim5Fault9HttpFault9ExceptionE(vim.fault.HttpFault)
--> [context]zKq7AVECAAAAAMJySwAUdnB4ZAAAkJ4rbGlidm1hY29yZS5zbwAAhE8bAOqXGAEOylR2cHhkAAH65ogBK+6IAbvs9QEchqIBJ3GiArgWAWxpYmF1dGh6Y2xpZW50LnNvAALYSAEC8E4BAsraAQJI0AEBhuWgAQrpVAE48lQBapVTA+AFAmxpYmMuc28uNgABdY1T[/context]
YYYY-MM-DDThh:mm:ss.msZ error vpxd[7F38CC164800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Failed to connect to IS: <N5Vmomi5Fault17HostCommunication9ExceptionE(vmodl.fault.HostCommunication)</font>
--> [context]zKq7AVECAAAAAMJySwATdnB4ZAAAkJ4rbGlidm1hY29yZS5zbwAAhE8bAOqXGAEOylR2cHhkAAF2E2IBsu/1ARyGogEncaICuBYBbGliYXV0aHpjbGllbnQuc28AAthIAQLwTgECytoBAkjQAQGG5aABCulUATjyVAFqlVMD4AUCbGliYy5zby42AAF1jVM=[/context]>
YYYY-MM-DDThh:mm:ss.msZ info vpxd[7F38CC164800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Retry for this error: attempt count 1
YYYY-MM-DDThh:mm:ss.msZ warning vpxd[7F38B6346700] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f385c24cb90, h:32, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:10080'>>, e: system:111(Connection refused)
YYYY-MM-DDThh:mm:ss.msZ warning vpxd[7F38CC164800] [Originator@6876 sub=VpxdAuthClient] [ConnectAndLogin] Failed to loginBySamlToken: N7Vmacore15SystemExceptionE(Connection refused)
--> [context]zKq7AVECAAAAAMJySwAPdnB4ZAAAkJ4rbGlidm1hY29yZS5zbwAAhE8bAOqXGADgXysAWVcjAApfJgD+YCYAhIMmAICKJgCfISQAb+0jAFbwIwB79isBVHQAbGlicHRocmVhZC5zby4wAAItjg5saWJjLnNvLjYA[/context]
YYYY-MM-DDThh:mm:ss.msZ info vpxd[7F38CC164800] [Originator@6876 sub=VpxdAuthClient] fallback to loginByCertificate
YYYY-MM-DDThh:mm:ss.msZ warning vpxd[7F38B75EB700] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f385c24cb90, h:32, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:10080'>>, e: system:111(Connection refused)
YYYY-MM-DDThh:mm:ss.msZ error vpxd[7F38CC164800] [Originator@6876 sub=httpUtil] [HttpUtil::ExecuteRequest] Error in sending request - Connection refused
YYYY-MM-DDThh:mm:ss.msZ error vpxd[7F38CC164800] [Originator@6876 sub=ServerAccess] Remote login failed: N3Vim5Fault9HttpFault9ExceptionE(vim.fault.HttpFault)
--> [context]zKq7AVECAAAAAMJySwAUdnB4ZAAAkJ4rbGlidm1hY29yZS5zbwAAhE8bAOqXGAEOylR2cHhkAAH65ogBK+6IAbvs9QEchqIBJ3GiArgWAWxpYmF1dGh6Y2xpZW50LnNvAALYSAEC8E4BAsraAQJI0AEBhuWgAQrpVAE48lQBapVTA+AFAmxpYmMuc28uNgABdY1T[/context]
YYYY-MM-DDThh:mm:ss.msZ error vpxd[7F38CC164800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Failed to connect to IS: <N5Vmomi5Fault17HostCommunication9ExceptionE(vmodl.fault.HostCommunication)</font>
--> [context]zKq7AVECAAAAAMJySwATdnB4ZAAAkJ4rbGlidm1hY29yZS5zbwAAhE8bAOqXGAEOylR2cHhkAAF2E2IBsu/1ARyGogEncaICuBYBbGliYXV0aHpjbGllbnQuc28AAthIAQLwTgECytoBAkjQAQGG5aABCulUATjyVAFqlVMD4AUCbGliYy5zby42AAF1jVM=[/context]>
 
...
 
YYYY-MM-DDThh:mm:ss.msZ error vpxd[7F38CC164800] [Originator@6876 sub=Authorize] Failed to initialize authorizeManager
YYYY-MM-DDThh:mm:ss.msZ warning vpxd[7F38CC164800] [Originator@6876 sub=VpxProfiler] Start [VpxdAuthorize::Start()] took 183263 ms
YYYY-MM-DDThh:mm:ss.msZ warning vpxd[7F38CC164800] [Originator@6876 sub=VpxProfiler] ServerApp::Start [TotalTime] took 183356 ms
YYYY-MM-DDThh:mm:ss.msZ error vpxd[7F38CC164800] [Originator@6876 sub=Default] Failed to start VMware VirtualCenter. Shutting down
YYYY-MM-DDThh:mm:ss.msZ info vpxd[7F38CC164800] [Originator@6876 sub=SupportMgr] Wrote uptime information
YYYY-MM-DDThh:mm:ss.msZ info vpxd[7F38BE15F700] [Originator@6876 sub=HostGateway] CmConnectionFSM::RunFSM(ST_CM_CALL_FAILED)
YYYY-MM-DDThh:mm:ss.msZ warning vpxd[7F38BE15F700] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Invalid argument : cmStub
YYYY-MM-DDThh:mm:ss.msZ warning vpxd[7F38BE15F700] [Originator@6876 sub=HostGateway] State(ST_ACQUIRE_TOKEN) failed with: Invalid argument : stsClient
YYYY-MM-DDThh:mm:ss.msZ info vpxd[7F38BE15F700] [Originator@6876 sub=HostGateway] stsUrlFromConfig: https://example-psc.example.com/sts/STSService/vsphere.local ssoAdminUrlFromConfig: https://example-psc.example.com/sso-adminserver/sdk/vsphere.local
YYYY-MM-DDThh:mm:ss.msZ info vpxd[7F38BE15F700] [Originator@6876 sub=vpxCrypt] Failed to read X509 cert; err: 151441516
YYYY-MM-DDThh:mm:ss.msZ info vpxd[7F38BE15F700] [Originator@6876 sub=vpxCrypt] Failed to read X509 cert; err: 151441516

  • In the /var/log/vmware/vmafdd/vmafdd-syslog.log files, you see entries similar to:
YYYY-MM-DDThh:mm:ss.ms.213900+00:00 err vmafdd t@139929541560064: VmAfSrvCfgGetMachineID failed. Error(9234)
YYYY-MM-DDThh:mm:ss.ms.214035+00:00 err vmafdd t@139929541560064: [Error - 9234, ../../../server/vmafd/config.c:440]
YYYY-MM-DDThh:mm:ss.ms.214166+00:00 err vmafdd t@139929541560064: VmAfSrvGetCMLocation failed. Error(9234)
YYYY-MM-DDThh:mm:ss.ms.214295+00:00 err vmafdd t@139929541560064: ERROR! [VmAfdIpcGetCMLocation] is returning [9234]
YYYY-MM-DDThh:mm:ss.ms.020892+00:00 err vmafdd t@139929716352768: [Error - 9234, ../../../server/vmafd/ldap.c:170]
YYYY-MM-DDThh:mm:ss.ms.021971+00:00 err vmafdd t@139929716352768: [Error - 9234, ../../../server/vmafd/rootfetch.c:256]
YYYY-MM-DDThh:mm:ss.ms.022297+00:00 notice vmafdd t@139929716352768: Failed to update trusted roots. Error [9234]
YYYY-MM-DDThh:mm:ss.ms.026505+00:00 err vmafdd t@139929564739328: [Error - 9234, ../../../server/vmafd/ldap.c:170]
YYYY-MM-DDThh:mm:ss.ms.704271+00:00 err vmafdd t@139929541560064: [Error - 9234, ../../../server/vmafd/vmdir.c:1468]
YYYY-MM-DDThh:mm:ss.ms.704528+00:00 notice vmafdd t@139929541560064: VmAfSrvDirOpenConnection failed. Error(9234)
YYYY-MM-DDThh:mm:ss.ms.704686+00:00 err vmafdd t@139929541560064: [Error - 9234, ../../../server/vmafd/vmdir.c:1023]
YYYY-MM-DDThh:mm:ss.ms.704825+00:00 notice vmafdd t@139929541560064: VmAfSrvDirGetMachineId failed. Error(9234)
YYYY-MM-DDThh:mm:ss.ms.704983+00:00 err vmafdd t@139929541560064: [Error - 9234, ../../../server/vmafd/config.c:1307]
  • When attempting to execute one of the VMAFD-CLI get-* or set-* commands, you see output similar to:
# /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
 
Error 9234: Authentication to VMware Directory Service failed.
Verify the username and password.
 
# /usr/lib/vmware-vmafd/bin/vmafd-cli set-dc-name --server-name localhost --dc-name example-vc.example.com
Error 9234: Authentication to VMware Directory Service failed.
Verify the username and password.
 
 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

 



Environment

  • VMware vCenter Server 5.5.x
  • VMware vCenter Server 6.x

Cause

Periodically, the password used by the Machine Accounts' is automatically refreshed.
 
This password is stored on the Machine Account on vCenter Server and within the VMware Directory Service residing on the Platform Services Controllers. If the backup used to perform a restore which pre-dates the most recent password refresh, upon restore the Machine Account will be unable to log into the VMware Directory Service.

Additionally, if the vCenter Server is rolled back to a snapshot state which pre-dates the most recent password refresh, upon restore the Machine Account will be unable to log into the VMware Directory Service.

Resolution


To update the password used by the vCenter Server's Machine Account after a restore, use the vcenter-restore CLI to issue a reset request. For more information, see the Restore the Failed vCenter Server Virtual Machine With the Direct-to-Host Emergency Restore Operation section in the vSphere Installation and Setup Guide.
 
For vCenter Server Appliance
 
  1. Login to vCenter Server via SSH as the root user
     
  2. Run this command to switch to BASH

    shell
     
  3. Run this command to Stop all of the vCenter Server services:

    service-control --stop --all
     
  4. Run this command (Below does not work on embedded node)

    vcenter-restore -u administrator -p <[email protected] password>

    For example:

    vcenter-restore -u administrator -p PASSWORD

    Note: This CLI can be executed without either option, and the user will be prompted to provide the administrator and [email protected] password.
  5. Run this command to confirm that all services are now started:

    service-control --status --all
 
For vCenter Server on Windows
 
  1. Open an elevated command prompt.
     
  2. Run this command to change the directories:

    cd C:\Program Files\VMware\vCenter Server\bin
     
  3. Run this command to Stop all of the vCenter Server services:

    service-control --stop --all
     
  4. Run this command (Below does not work on embedded node)

    vcenter-restore -u administrator -p <[email protected] password>

    For example:

    vcenter-restore -u administrator -p PASSWORD

    Note: This CLI can be executed without either option, and the user will be prompted to provide the administrator and [email protected] password.
     
  5. Run this command to confirm that all services are now started:

    service-control --status --all



Additional Information

To manually invoke an update of vCenter Server's Machine Account password, use the dir-cli computer password-reset CLI to issue a reset request.

 
For vCenter Server Appliance
 
  1. Login to the vCenter Server via SSH as the root user
     
  2. Run this command to switch to BASH

    shell
     
  3. Run this command:

    /usr/lib/vmware-vmafd/bin/dir-cli computer password-reset --login administrator --live-dc-hostname <Platform Services Controller FQDN> --password <[email protected] password>

    For example:

    /usr/lib/vmware-vmafd/bin/dir-cli computer password-reset --login administrator --live-dc-hostname example-psc.example.com --password 'PASSWORD'

    A successful password reset will output:

    Password for machine account reset.

    Note: If you are using a PSC HA configuration using a Load Balancer, execute this command against one of the backing PSC members and wait 30 seconds for the VMware Directory Service replication cycle to push this update to all nodes.
     
  4. Run these commands to Start the vCenter Server services:

    service-control --stop --all
    service-control --start --all
Powered by Wolken Software