vCenter services fails to start and pointing to an incorrect Platform Services Controller

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vCenter services fails to start and pointing to an incorrect Platform Services Controller
• Upgrades that include a PSC convergence might fail.
• service-control fails with entries similar to:

Error Failed to start vmon services.vmon-cli RC=1, stderr=Failed to start sps, content-library, vsm, vsan-health, rbd, statsmonitor services. Error: Operation timed out

• In vmafdd.log log, you see entries similar to:

18-07-06T16:32:36.418368-04:00 notice vmafdd  t@140651615987456: VECS_IPC_OPEN_CERTSTORE: elasped time 60 seconds
18-07-06T16:32:51.177795-04:00 notice vmafdd  t@140651632772864: VECS_IPC_OPEN_CERTSTORE: elasped time 3 seconds
18-07-06T16:32:51.201577-04:00 notice vmafdd  t@140651624380160: VECS_IPC_OPEN_CERTSTORE: elasped time 19 seconds
18-07-06T16:32:52.165775-04:00 notice vmafdd  t@140651807729408: Affinitized to [psc2.vmware.domain]. Previously affinitized to [psc1.vmware.domain]
18-07-06T16:32:52.166119-04:00 err vmafdd  t@140651807729408: [Error - 21, ../../../server/vmafd/superlogging.c:535]
18-07-06T16:32:52.171947-04:00 err vmafdd  t@140651807729408: [Error - 21, ../../../server/vmafd/superlogging.c:535]
18-07-06T16:32:52.292763-04:00 notice vmafdd  t@140651615987456: PSC [psc2.vmware.domain] is not Alive for requests
18-07-06T16:33:52.299056-04:00 err vmafdd  t@140651607594752: [Error - 5, ../../../server/db/cdcdbutil.c:458]
18-07-06T16:33:52.299332-04:00 err vmafdd  t@140651607594752: [Error - 5, ../../../server/vmafd/cdcservice.c:435

• In vmafd/parameter of likewise shell, you see a DCNameHA value:

Example: 
# /opt/likewise/bin/lwregshell ls "[HKEY_THIS_MACHINE\Services\vmafd\Parameters]" | grep DCName | awk '{print $2,$NF}'
"DCName" "psc1.vmware.domain"
"DCNameHA" "psc2.vmware.domain"

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server Appliance 6.5.x

Cause

This issue occurs when a stale parameter "DCNameHA" exists in vmafd\parameters, causing vCenter to use the node specified as its PSC URL.

Resolution

The cause for these stale entries has been resolved in vCenter Server 6.5 Update 2 and later. However, if the stale DCNameHA entry already exists in an environment, it will need to be removed with the steps in the workaround section below.

Workaround:

1. To review the status of client-affinity state from a vCenter node:

 

/usr/lib/vmware-vmafd/bin/cdc-cli client-affinity state

Output:
Client Domain Controller's cache's current state:        Site affinitized

Note :  If there are no PSC HA configured, then the state would be Legacy Mode

       

2. To change the state:

/usr/lib/vmware-vmafd/bin/cdc-cli client-affinity legacy
 

3. Validate the  DCNameHA removed from vmafd/parameters :

/opt/likewise/bin/lwregshell ls "[HKEY_THIS_MACHINE\Services\vmafd\Parameters]" | grep DCName | awk '{print $2,$NF}'

4. Start Services

How to stop, start, or restart vCenter Server 6.x services
Stopping, starting, or restarting services in vCenter Server Appliance 6.5