vCenter Server services fails to start and pointing to an incorrect Platform Services Controller

Products

• VMware vCenter Server

Issue/Introduction

• vCenter Server services fails to start and pointing to an incorrect Platform Services Controller
• Upgrades that include a PSC convergence might fail.
• service-control --start --all command fails with entries similar to:

Error Failed to start vmon services.vmon-cli RC=1, stderr=Failed to start sps, content-library, vsm, vsan-health, rbd, statsmonitor services. Error: Operation timed out

• In vmafdd.log log, you see entries similar to:
  Log Location - /var/log/vmware/vmafdd/vmafdd.log - 

YYYY-MM-DDTHH:MM:SS notice vmafdd  t@140651615987456: VECS_IPC_OPEN_CERTSTORE: elasped time 60 seconds
YYYY-MM-DDTHH:MM:SS notice vmafdd  t@140651632772864: VECS_IPC_OPEN_CERTSTORE: elasped time 3 seconds
YYYY-MM-DDTHH:MM:SS notice vmafdd  t@140651624380160: VECS_IPC_OPEN_CERTSTORE: elasped time 19 seconds
YYYY-MM-DDTHH:MM:SS notice vmafdd  t@140651807729408: Affinitized to [psc2.domain.com]. Previously affinitized to [psc1.domain.com]
YYYY-MM-DDTHH:MM:SS err vmafdd  t@140651807729408: [Error - 21, ../../../server/vmafd/superlogging.c:535]
YYYY-MM-DDTHH:MM:SS err vmafdd  t@140651807729408: [Error - 21, ../../../server/vmafd/superlogging.c:535]
YYYY-MM-DDTHH:MM:SS notice vmafdd  t@140651615987456: PSC [psc2.domain.com] is not Alive for requests
YYYY-MM-DDTHH:MM:SS err vmafdd  t@140651607594752: [Error - 5, ../../../server/db/cdcdbutil.c:458]
YYYY-MM-DDTHH:MM:SS err vmafdd  t@140651607594752: [Error - 5, ../../../server/vmafd/cdcservice.c:435

• In vmafd/parameter of likewise shell, you see a DCNameHA value:

Example: 

# /opt/likewise/bin/lwregshell ls "[HKEY_THIS_MACHINE\Services\vmafd\Parameters]" | grep DCName | awk '{print $2,$NF}'
"DCName" "psc1.domain.com"
"DCNameHA" "psc2.domain.com"

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server Appliance 6.5.x

Cause

This issue occurs when a stale parameter "DCNameHA" exists in vmafd\parameters, causing vCenter to use the node specified as its PSC URL.

Resolution

The cause for these stale entries has been resolved in vCenter Server 6.5 Update 2 and later. However, if the stale DCNameHA entry already exists in an environment, it will need to be removed with the steps in the workaround section below.

Workaround:

1. To review the status of client-affinity state from a vCenter node:

/usr/lib/vmware-vmafd/bin/cdc-cli client-affinity state

Output:

Client Domain Controller's cache's current state:        Site affinitized

Note :  If there are no PSC HA configured, then the state would be Legacy Mode

       

2. To change the state:

/usr/lib/vmware-vmafd/bin/cdc-cli client-affinity legacy
 

3. Validate the  DCNameHA removed from vmafd/parameters :

/opt/likewise/bin/lwregshell ls "[HKEY_THIS_MACHINE\Services\vmafd\Parameters]" | grep DCName | awk '{print $2,$NF}'

4. Restart Services on vCenter Server: 
   
   service-control --stop --all && service-control --start --all 

Reference : Stopping, starting, or restarting services in vCenter Server Appliance 6.5 and beyond