"Cannot load the users for the selected domain / Error while extracting local SSO users", Unable to add Active Directory users or groups to vCenter Server Appliance or vRealize Automation permissions
search cancel

"Cannot load the users for the selected domain / Error while extracting local SSO users", Unable to add Active Directory users or groups to vCenter Server Appliance or vRealize Automation permissions

book

Article ID: 322244

calendar_today

Updated On:

Products

VMware Aria Suite VMware vCenter Server

Issue/Introduction

When using the Active Directory (Integrated Windows Authentication) identity source from the vCenter Single Sign-On 5.5 (SSO), Platform Services Controller 6.0 (PSC), or vRealize Automation Identity Appliance, you experience these symptoms:
  • Attempting to browse and add users to the vCenter Server permissions (Local Permission: Hosts and Clusters > vCenter > Manage > Permissions, Global Permissions: Administration > Global Permissions) fails with one of below errors:
Cannot load the users for the selected domain.
OR
Error while extracting local SSO users  
  • Attempting to browse users from your Active Directory Domain under the Users tab (Administration > Users and Groups) in the vCenter Server fails with the error:

    com.vmware.identity.idm.IDMException: Failed to establish server connection.
     
  • Attempting to browse and add users to the vRealize Automation Center permissions fails with the error:

    System Exception.
     
  • In the /var/log/vmware/sso/vmware-sts-idmd.log file on the vCenter Single Sign-On or Platform Services Controller, you see entries similar to:

    [YYYY-MM-DDT<time>Z vsphere.local 3572c5f8-####-####-####-d94f68634a2f WARN ] [ServerUtils] cannot bind connection: [ldap://<Active Directory Domain Controller FQDN>, null]</time>
    [YYYY-MM-DDT<time>Z vsphere.local 3572c5f8-####-####-####-d94f68634a2f ERROR] [ServerUtils] cannot establish connection with uri: [ldap://<Active Directory Domain Controller FQDN>]</time>
    [YYYY-MM-DDT<time>Z vsphere.local 3572c5f8-####-####-####-d94f68634a2f INFO ] [ActiveDirectoryProvider] removeDcInfo - domain [<Active Directory Domain Name>], domainFQDN [<Active Directory Domain Controller FQDN>], domainIpAddress [<Active Directory Domain Controller IP]</font></time>
    [YYYY-MM-DDT<time>Z vsphere.local 3572c5f8-####-####-####-d94f68634a2f ERROR] [ActiveDirectoryProvider] Failed to get non-GC connection to domain <Active Directory Domain> - domain controller might be offline</time>
    com.vmware.identity.interop.idm.IdmNativeException: Native platform error [code: 40287][LW_ERROR_LDAP_LOCAL_ERROR][]
    at com.vmware.identity.interop.idm.LinuxIdmNativeAdapter.LdapSaslBind(LinuxIdmNativeAdapter.java:345)
    at com.vmware.identity.interop.ldap.LinuxLdapClientLibrary.ldap_sasl_bind_s(LinuxLdapClientLibrary.java:676)
    at com.vmware.identity.interop.ldap.LdapConnection.bindSaslConnection(LdapConnection.java:158)
    at com.vmware.identity.idm.server.ServerUtils.getLdapConnection(ServerUtils.java:297)
    at com.vmware.identity.idm.server.ServerUtils.getLdapConnectionByURIs(ServerUtils.java:215)

    ...

    [YYYY-MM-DDT<time>Z</time> vsphere.local b77dc08e-####-####-####-eee92feae7c6 WARN ] [ServerUtils] cannot bind connection: [ldap://<Active Directory Domain Controller FQDN>, null]
    [YYYY-MM-DDT<time>Z</time> vsphere.local b77dc08e-####-####-####-eee92feae7c6 ERROR] [ServerUtils] cannot establish connection with uri: [ldap://<Active Directory Domain Controller FQDN>]
    [YYYY-MM-DDT<time>Z</time> vsphere.local b77dc08e-####-####-####-eee92feae7c6 ERROR] [ActiveDirectoryProvider] Failed to get non-GC connection to domain <Active Directory Domain Name> in retry
    com.vmware.identity.interop.idm.IdmNativeException: Native platform error [code: 40287][LW_ERROR_LDAP_LOCAL_ERROR][]
    YYYY-MM-DDT<time>Z</time> vsphere.local b77dc08e-####-####-####-eee92feae7c6 ERROR] [IdentityManager] Failed to find person users [Criteria : searchString=, domain=markit.partners] in tenant [vsphere.local]
    [YYYY-MM-DDT<time>Z</time> vsphere.local b77dc08e-####-####-####-eee92feae7c6 ERROR] [ServerUtils] Exception 'com.vmware.identity.idm.IDMException: Failed to establish server connection'
    com.vmware.identity.idm.IDMException: Failed to establish server connection
    ...
    Caused by: com.vmware.identity.idm.IDMException: Failed to get non-GC connection to domain <Active Directory Domain Name> in retry
    [YYYY-MM-DD <time> vsphere.local 9439b581-####-####-####-39d3af448747 WARN ] [ServerUtils] cannot bind connection: [ldap://</time><Active Directory Domain Controller FQDN>, null]
    [YYYY-MM-DD <time> vsphere.local 9439b581-####-####-####-39d3af448747 ERROR] [ServerUtils] cannot establish connection with uri: [ldap://</time><Active Directory Domain Controller FQDN>]
    [YYYY-MM-DD <time> vsphere.local 9439b581-####-####-####-39d3af448747 ERROR] [IdentityManager] Failed to find person users [Criteria : searchString=po.tenant, domain=<Active Directory Domain Name>] in tenant [vsphere.local] </time>
    [YYYY-MM-DD <time> vsphere.local 9439b581-####-####-####-39d3af448747 ERROR] [ServerUtils] Exception 'com.vmware.identity.idm.IDMException: Failed to establish server connection' </time>
    com.vmware.identity.idm.IDMException: Failed to establish server connection
    ... 22 more

    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.


Environment

VMware vCenter Server Appliance 6.0.x
VMware vCenter Server Appliance 5.5.x
VMware vCenter Server Appliance 6.5.x
VMware vCloud Automation Center for Desktop 6.1.x
VMware vRealize Automation 6.2.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server 7.0.x
VMware vCloud Automation Center for Desktop 6.0.x

Cause

This issue occurs because the Likewise Kerberos stack requires all DNS servers to be configured with the Reverse Lookup Zone and that all Active Directory Domain Controller (AD DC) Pointer (PTR) records are available. The Likewise Kerberos stack in the Appliances use both Forward and Reverse Name Lookup to canonically organize hostnames for use in service principal names.

Resolution

To resolve this issue, ensure that all DNS servers have the Reverse Lookup Zone configured and Active Directory Domain Controller (AD DC) Pointer (PTR) records present.

Determining the DNS servers of vCenter Server or vRealize Automation Appliance

  1. Initiate an SSH connection to the vCenter Server or vRealize Automation Appliance.
  2. Enter the root username and password when prompted.

    Note: If you are using vSphere 6.0, run these commands to switch to the Bash shell:

    shell.set --enable True
    shell
     
  3. Run this command to review the DNS servers configured for the vCenter Server or vRealize Automation Appliance:

    less /etc/resolv.conf

    For example:

    nameserver 10.100.10.213
    nameserver 10.10.10.252
Note: On VCSA 6.5/6.7, you will see another nameserver entry (nameserver 127.0.0.1). This entry is for the local DNS cache served by Dnsmasq Service.

 

Checking Active Directory Trust Enumeration

To determine all trusts that are enumerated by the SSO 5.5, PSC 6.0, or Identity Appliance 6.x:
  1. Initiate an SSH connection to the SSO, PSC, or Identity Appliance.
  2. Enter the root user name and password when prompted.

    Note: If using vSphere 6.0, run the following command to switch to the Bash shell:

    shell.set --enable True
    shell
     
  3. Run this command to review all of the enumerated trusts from the Likewise Kerberos stack on the SSO, PSC, or Identity Appliance Appliance:

    less /var/lib/likewise/krb5-affinity.conf

    Note: This will output all of the trusts currently accessible from the SSO, PSC, or Identity Appliance.

    You see output similar to:

    [realms]

    DomainA.local = {
    kdc = 10.10.10.213
    }
    DomainB.local = {
    kdc = 10.10.10.81
    }
    ChildDomainA.DomainB.local = {
    kdc = 10.10.10.85
    }
    ChildDomainB.DomainB.Local = {
    kdc = 10.10.10.83
    }
    DomainC.local = {
    kdc = 10.10.10.252
    kdc = 10.10.10.250
    }
    ChildDomainC.DomainB.local = {
    kdc = 10.10.10.247
    kdc = 10.10.10.82
    }
     
  4. Run this command to view a list of domain controllers that are not accessible from the Appliance:

    grep "cannot establish connection with uri:" /var/log/vmware/sso/vmware-sts-idmd.log | cut -d'[' -f4 | sort -nr | uniq -c

    Or

    grep "cannot establish connection with uri:" /var/log/vmware/sso/vmware-sts-idmd.log | cut -d'[' -f4 | uniq

    Note: In 6.5 vCenter Server, try the grep command grep "cannot establish connection with uri:" /var/log/vmware/sso/vmware-sts-idmd.log | cut -d']' -f 3 | cut -d '/' -f 3 | sort | uniq -c
    if above mentioned commands are not listing any DC names

    Note: In vCenter 7 version and up run the command  grep -i " cannot establish" /var/log/vmware/sso/* | cut -d']' -f 3-10 | sort | uniq -c

    You see output similar to:

    ldap://localhost:389]
    ldap://dc2-root.DomainA.local]
    ldap://Vigrid.local]
    ldap://DC-4.DomainB.local]
    ldap://dc-us.DomainC.local]
    ldap://dc2-nh.DomainB.local]
    ldap://sqa-dc-3.DomainB.local]
    ldap://dc2-root.DomainA.local]
    ldap://DC-4.DomainB.local]

Checking Active Directory Domain Controller DNS Resolution:

  1. Initiate an SSH connection to the Appliance.
  2. Enter the root username and password when prompted.

    Note: If you are using vSphere 6.0, run this command to switch to the Bash shell:

    shell.set --enable True
    shell
     
  3. Using nslookup from the Appliance, run this command to ensure there is DNS resolution for Forward Lookup for the Domain Controllers determined from the Checking Trust Enumeration section:

    nslookup dc2-root.DomainA.local

    Note: This command displays the IP address of the Domain Controller.

    You see output similar to:

    nslookup dc2-root.DomainB.local
    Server: 10.100.10.213
    Address: 10.100.10.213#53

    Non-authoritative answer:
    Name: dc2-root.DomainB.local
    Address: 10.10.10.81
     
  4. To ensure that there is DNS resolution for Reverse Lookup for the domain controllers, run this command:

    nslookup 10.10.10.81

    If the Reverse Lookup is incorrect or missing (there are chances of multiple incorrect PTR records as well), you will see output similar to:

    nslookup 10.10.10.81
    Server: 10.100.10.213
    Address: 10.100.10.213#53

    Non-authoritative answer:
    81.10.10.10.in-addr.arpa name = <Incorrect FQDN>.

    Authoritative answers can be found from:
     
  5. Repeat Steps 1 to 4 for any additional Active Directory Domain Controllers to determine the records that are missing or incorrect.
     
  6. To resolve the issue when there are missing or incorrect records, use one of these options:
    • Option 1: Create or update the PTR record(s) for the Active Directory Domain Controller(s) on the listed DNS Servers from the Determining the Appliance's DNS Servers section.
    • Option 2: Update the DNS servers configured on the appliance to use DNS servers containing the correct PTR for your Active Directory Domain Controllers records. For more information, see Edit the DNS and IP Address Settings of the vCenter Server Appliance section in the vCenter Server Appliance Configuration guide.
    • Option 3: Add the missing Reverse Lookup records for the Active Directory Domain Controller(s) to the Appliance's /etc/hosts. For more information, see Editing files on an ESX host using vi or nano (1020302).

      Entries added to /etc/hosts file on the Appliance should be in the following format:

      IP_Address FQDN_of_Domain_Controller Short_Name_of_Domain_Controller

      For example:

      10.10.10.81 dc2-root.DomainB.local dc2-root



Additional Information

Powered by Wolken Software