Unable to link HLM: Link failed with reason: Internal system error. Retry

Products

• VMware Cloud on AWS

Issue/Introduction

Symptoms:

• You can see this issue while trying to Re-link HLM
• Error Message: Link failed with reason: Internal system error. Retry
• In the hvc logs you see the below messages:
  • [tomcat-exec-7  INFO  com.vmware.hvc.setup.SolutionUser  opId=] ReadOnly already assigned to hvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    [tomcat-exec-7  INFO  com.vmware.hvc.setup.SolutionUser  opId=] Permissions added to the solution user - hvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    [tomcat-exec-7  INFO  com.vmware.hvc.vapi.impl.LinksProviderImpl  opId=] Step 6: Importing webclient solution user on foreign.
    [tomcat-exec-7  WARN  com.vmware.vim.sso.admin.client.vmomi.impl.AbstractClient  opId=] ******* WARNING ****** WARNING ****** WARNING *******
    [tomcat-exec-7  WARN  com.vmware.vim.sso.admin.client.vmomi.impl.AbstractClient  opId=] Possible remote API mismatch detected. Operation will continue, but errors are likely.
    [tomcat-exec-7  WARN  com.vmware.vim.sso.admin.client.vmomi.impl.AbstractClient  opId=] ******* WARNING ****** WARNING ****** WARNING *******
    [tomcat-exec-7  INFO  com.vmware.identity.token.impl.SamlTokenImpl  opId=] SAML token for SubjectNameId [[email protected], format=] successfully parsed from Element
    [tomcat-exec-7  WARN  com.vmware.vim.sso.admin.client.vmomi.impl.AbstractClient  opId=] ******* WARNING ****** WARNING ****** WARNING *******
    [tomcat-exec-7  WARN  com.vmware.vim.sso.admin.client.vmomi.impl.AbstractClient  opId=] Possible remote API mismatch detected. Operation will continue, but errors are likely.
    [tomcat-exec-7  WARN  com.vmware.vim.sso.admin.client.vmomi.impl.AbstractClient  opId=] ******* WARNING ****** WARNING ****** WARNING *******
    [tomcat-exec-7  INFO  com.vmware.hvc.setup.SolutionUser  opId=] vsphere-webclient-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx already exists.
    [tomcat-exec-7  INFO  com.vmware.hvc.lotus.LdapUtil  opId=] Remove remote site DN:,cn=ForeignDomains,cn=Configuration,dc=vmc,dc=local
    tomcat-exec-7  INFO  com.vmware.hvc.vapi.impl.LinksProviderImpl  opId=] Cleaned up LS registrations
    [tomcat-exec-7  INFO  com.vmware.hvc.vapi.impl.LinksProviderImpl  opId=] Removed join file xxxxxxxxxxx
    [tomcat-exec-7  ERROR com.vmware.hvc.vapi.impl.LinksProviderImpl  opId=] Link failed with reason: Internal system error. Retry
    com.vmware.vim.sso.admin.exception.InvalidPrincipalException: The specified principal (Principal {Name: vsphere-webclient-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, Domain: vmc.local} is not a user) is invalid.
    at com.vmware.vim.sso.admin.client.vmomi.impl.VmomiClientCommand.execute(VmomiClientCommand.java:124)
    at java.lang.Thread.run(Thread.java:748)
    [tomcat-exec-7  INFO  com.vmware.hvc.vapi.impl.LinksProviderImpl  opId=] Duration of creating link is 16080 ms
    [tomcat-exec-7  INFO  com.vmware.hvc.vapi.impl.LinksProviderImpl  opId=] Pushing create link data to VAC
    [tomcat-exec-7  INFO  com.vmware.hvc.vapi.impl.LinksProviderImpl  opId=] Successfully sent item: {"@type":"hlm_events","@id":"xxxxxxxxxxxb","item_type":"Link","foreign_id":"xxxxxxxxxxxx","status":"Link failed with reason: Internal system error. Retry","duration":"16080"} to url:

Cause

As part of HLM Linking, we copy the hvc & vsphere-webclient solution users from the cloud to on-prem. 
This issue can be experienced if you have left over solution users from a previous linking attempt.

Resolution

We do not have a permanent fix for this issue yet. Please refer to the workaround section.

Workaround:

1. Ensure to take a snapshot of the on-premises vCenter Server Appliance VM.
2. Open On-Prem VC which is participating in Hybrid Linked Mode with the cloud vCenter.
3. Navigate to Home > Administration > SSO > Users & Groups > Solution Users
4. You should see one solution user here which have been copied over from the cloud VC.

You should see something like:

hvc-xx-xx-xx-xx

4. Click on it and then press the red cross to delete this solution user.
5. Then retry the linking UI workflow again via "Home > "Administration" > "Hybrid Cloud: Linked Domains" input user parameters and linking should be successful.

 Note: This solution user will have the same Valid To date and if you are unsure which solution user to delete, refer to the hvc logs or reach out to VMware support for help.