vCenter linking in VCF Operations fails with errors, "An error has occurred. Retry the linking process after sometime." and "There are no valid vcenter instances available to proceed with the linking process."

Products

VCF Operations VMware vCenter Server

Issue/Introduction

While attempting to create vCenter linking group in VCF Operations, the following errors are encountered, "An error has occurred. Retry the linking process after sometime." , “There are no valid vCenter instances available to proceed with the linking process.”
The /storage/vcops/log/vrops-vcenter-linking.log on VCF Operations shows entries similar to:
YYYYY-MM-DDThh:mm:ss ERROR [AsyncExecutor-2] com.vmware.vrops.vcenterlinking.manager.StateManager - TRUST_ESTABLISHMENT failed for the Vcenter Id: 84#####-####-####-####-#########69a with reason: ERROR_UNKNOWN
YYYYY-MM-DDThh:mm:ss INFO [AsyncExecutor-2] com.vmware.vrops.vcenterlinking.manager.StateManager - Starting TRUST_ESTABLISHMENT state processing for vCenter c6#####-####-####-####-#########e9c
YYYYY-MM-DDThh:mm:ss INFO [AsyncExecutor-2] com.vmware.vrops.vcenterlinking.manager.StateManager - Initiating FetchIdentityParam task for vCenter c6#####-####-####-####-#########e9c
YYYYY-MM-DDThh:mm:ss INFO [AsyncExecutor-2] com.vmware.vrops.vcenterlinking.manager.StateManager - Updating vCenter c6#####-####-####-####-#########e9c with Domain Id ca#####-####-####-####-#########e8e
YYYYY-MM-DDThh:mm:ss ERROR [AsyncExecutor-2] com.vmware.vrops.vcenterlinking.manager.StateManager - TRUST_ESTABLISHMENT failed for the Vcenter Id: c68#####-####-####-####-#########e9c with reason: NO_AVAILABLE_PEERS
The /storage/vcops/log/adapters/ManagementAdapter/ManagementAdapter_xx.log on VCF Operations (or Cloud proxy) shows
YYYYY-MM-DDThh:mm:ss ERROR ManagementAdapter 3477974 [ops@4413 threadId="46406" threadName="TasksManager-TaskHandler-545" instanceId="512"] [(512) com.vmware.adapter.management.components.vclinking.task.C.N] - Exception while establishTrust
com.vmware.vapi.std.errors.AlreadyExists: AlreadyExists (com.vmware.vapi.std.errors.already_exists) => {
messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
id = com.vmware.vcenter.trustmanagement.alreadyexists,
defaultMessage = Trust object already exists for given domain ID (ca#####-####-####-####-#########e8e),
args = [ca#####-####-####-####-#########e8e],
params = <null>,
localized = <null>
}],
data = <null>,
errorType = ALREADY_EXISTS
The domainGUID on multiple vCenter systems in the vCenter linking group have the same domainGUID output verified in a ssh session to each vCenter as root with the command:
root@<vCenter> [ ~ ]# /opt/likewise/bin/lwregshell list_values '[HKEY_THIS_MACHINE\Services\vmafd\Parameters]' | grep -i DomainGUID
+ "DomainGUID" REG_SZ "ca#####-####-####-####-#########e8e"
Attempting to run break-elm as outlined in Deactivating Enhanced Linked Mode (ELM) in vCenter 9.0 (VVF) fails with an error similar to:
YYYYY-MM-DDThh:mm:ss INFO cmsso_util Errors: [
YYYYY-MM-DDThh:mm:ss INFO cmsso_util {hostname: <vCenterFQDN>, error: Cannot invoke "String.equalsIgnoreCase(String)" because the return value of "java.net.URI.getHost()" is null}
YYYYY-MM-DDThh:mm:ss INFO cmsso_util ]

Environment

VCF 9.0.x
vCenter 9.0.x

Cause

This issue occurs when the vCenter instances have had Enhanced Linked Mode (ELM) broken prior to upgrading to 9.0, but the domainGUID was not updated.

Resolution

Ensure you have offline snapshots of all vCenter servers that were previously connected with ELM and you intend to link. Also ensure you have a valid backup. Do not skip this step.
ssh as the root user (switch to bash shell if needed) to one of the vCenter systems with the duplicate DomainGUID that you intend to link with vCenter linking.
Generate a new UUID using this command and save the generated ID:
python -c "import uuid; print(str(uuid.uuid4()))"
Run the following command and replace <GUID HERE> with the ID generated in the previous step. Note: You will need the SSO administrator user [email protected] account password to complete this step. If the vCenter is using a custom SSO domain, replace vsphere.local with the custom domain name for the admin account and all CN entries below.
TIP: Use notepad to prepare the ldapmodify command. (The 6 separate lines below will need to be copied and pasted as a whole and not separate lines (and no spaces in-between lines) into the SSH session)
/opt/likewise/bin/ldapmodify -x -D cn=Administrator,cn=Users,dc=vsphere,dc=local -W <<EOF
dn: dc=vsphere,dc=local
changetype: modify
replace: objectGUID
objectGUID: <GUID HERE>
EOF
Check that the ldap modify command was successful with the following command:
/opt/likewise/bin/ldapsearch -b "dc=vsphere,dc=local" -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -s sub "(|(objectclass=dcObject))" objectGUID -W

Sample output: If the command was successful, the highlighted objectGUID should be the new GUID provided and not the original duplicate ID.

root@<vCenter> [ ~ ]# /opt/likewise/bin/ldapsearch -b "dc=vsphere,dc=local" -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -s sub "(|(objectclass=dcObject))" objectGUID -W
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <dc=vsphere,dc=local> with scope subtree
# filter: (|(objectclass=dcObject))
# requesting: objectGUID
#

# vsphere.local
dn: dc=vsphere,dc=local
objectGUID: #######-####-####-####-############

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1
Update likewise with the following command:
/opt/likewise/bin/lwregshell set_value "[HKEY_THIS_MACHINE\\Services\\vmafd\\Parameters]" "DomainGUID" "<GUID HERE>"
Verify the domainGUID was updated in lwregshell with:
/opt/likewise/bin/lwregshell list_values '[HKEY_THIS_MACHINE\Services\vmafd\Parameters]' | grep -i DomainGUID
Sample output:
root@<vCenter> [ ~ ]# /opt/likewise/bin/lwregshell list_values '[HKEY_THIS_MACHINE\Services\vmafd\Parameters]' | grep -i DomainGUID
+ "DomainGUID" REG_SZ "#######-####-####-####-############"
Restart vCenter services:
service-control --stop --all && service-control --start --all
Return to VCF Operations, remove the failed vCenter linking group and re-create the VCF Linking group.

Additional Information

vCenter linking in VCF Operations fails with error