VMware Cloud on AWS troubleshooting Hybrid Linked Mode permission issues

Products

• VMware Cloud on AWS

Issue/Introduction

The purpose of this article is to troubleshoot Hybrid Linked Mode related permission issues in VMware Cloud on AWS, after the SSO domain linking is successfully configured.

Symptoms:

• On-Prem vCenter is missing from VMware Cloud on AWS HTML5 vSphere client inventory, even when the Linked Domains status is Domain linked

• Inconsistent behavior in workflows related to HLM.
  
  Example:
  
  Unable to proceed with cold migration with the console error:
  
  “The object 'vim.ResourcePool:resgroup-55' has already been deleted or has not been completely created”

Cause

This issue occurs for these reasons:

• The domain user account (or the domain group the user is part of) used to login to the cloud vCenter does not have proper permissions on the on-prem vCenter.
• SSO group membership based permissions in the on-prem vCenter does not work with HLM.
• HLM may no longer work if the current solution users are removed.
• Solution users from previous linking attempts were not removed properly.

Resolution

To resolve this issue, complete the following steps that are applicable:

1. Login directly to the on-prem vCenter to confirm the user privileges. If the issue is due to improper permissions for the logged in user, provide appropriate permissions:
   1. Login to the on-prem vSphere client
   2. Navigate to Administration > Access Control > Global Permissions
   3. Add users to one of the groups present already with correct roles assigned, or add new group with appropriate roles
2. Assign the domain group with proper roles as in step1, even if the user is already part of SSO groups such as Administrators. Cloud vCenter does not recognize permissions provided through the SSO group membership.
   
   Note: Do not remove these solution users used by HLM from the vCenter:

• syncservice-<machine-id-of-vmc> (administrative LDAP user)
• hvc-<machine-id-of-vmc> (solution user, same name as the one HVC service uses on the cloud)
• vsphere-webclient-<machine-id-of-vmc> (solution user, same name as the one H5 client uses on the cloud)
  
  The machine-id can be confirmed from cloud vSphere H5 client – Global Permissions, where the id is appended to these mentioned user names.

1. Unlinking the linked domain cleans up the related entries including the solution users. However, the user accounts are left behind in certain cases such as deleting SDDC without unlinking the domain first.
   
   Ensure to unlink HLM before deleting a SDDC:
   
   Cloud H5 client > Administration > Hybrid Cloud > Link Domains > Click the unlink button
   
   Caution: Removing incorrect solution users might result in issues.
2. If there are old solution users and web client users in the on-prem vCenter left behind by previous un-successful HLM configuration:
   1. Login to the on-prem vSphere client > Navigate to Administration > Access Control > Global Permissions.
   2. Delete any permissions you see for hvc-<unrecognizable machine id> and vsphere-webclient-< unrecognizable machine id
   3. Navigate to Administration > Single Sign On > Users and Groups > Solution Users
   4. Delete the old solution user
   5. Navigate to Administration > Single Sign On > Users and Groups > Users
   6. Delete the old sync user syncservice-<unrecognizable machine id>

Additional Information

Impact/Risks:
Removing incorrect solution users might result in issues, affecting other services