Unable to login to a vCenter Gateway using SSO domain credentials after upgrading to vCenter 7.0.

Products

• VMware vCenter Server 7.0

Issue/Introduction

During an upgrade of a vCenter Cloud Gateway to any release of vCenter 7, gateway logins using the system domain may fail under certain conditions. The system domain is typically vsphere.local.  This problem typically affects servers in Enhanced Link Mode groups, but may also affect an individual vCenter deployed with Cloud Gateway.

This KB article is strictly for fixing the vCenter Cloud Gateway Appliance. For information about repairing a vCenter Server Appliance, see KB 367229

Environment

vCenter Server 7.0.x

Cause

During the upgrade, the system domain's identity source is redirected to the system domain of the most recently installed server.  If that server is immediately taken offline, then NO servers can be logged into using the system domain identity source.  This problem occurs when the vCenter is taken offline during the upgrade process before completing it.  Note that upgrades irrevocably upgrade the schema, so once an upgrade is started, it must be finished expediently.

Resolution

To avoid this situation, upgrade both vCenter and Cloud Gateway before taking either of them offline.  If this cannot be avoided, use the workaround below to update the vmwSTSConnectionStrings value on the Cloud Gateway appliance.

Preparation

1. Connect to the Cloud Gateway via SSH.

2. Construct the LDAP distinguished name for the Administrator account.  For example, if the system domain is vsphere.local, the distinguished name is:

cn=Administrator, cn=Users, dc=vsphere, dc=local

3. Construct the LDAP distinguished name for the system domain configuration entry.  For example, if the system domain is vsphere.local, the distinguished name is:

cn=vsphere.local,cn=IdentityProviders,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local

4. Display the system domain definition on the local machine by entering the following command, then providing the password when prompted:

ldapsearch -h localhost -D "<Administrator LDAP Distinguished Name>" -W -b "<System domain configuration entry LDAP distinguished name>" -s base

For example, for the default domain of vsphere.local, the command is:

ldapsearch -h localhost -D "cn=Administrator, cn=Users, dc=vsphere, dc=local" -W -b "cn=vsphere.local,cn=IdentityProviders,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local" -s base

 

Repair

1. Scan the output, verifying that the vmwSTSConnectionStrings attribute is either equal to localhost, or to the FQDN of a vCenter taken offline.  If so, continue.

dn: cn=vsphere.local,cn=IdentityProviders,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local
objectClass: top
objectClass: vmwSTSIdentityStore
cn: vsphere.local
nTSecurityDescriptor:: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAA
vmwSTSIdentityStoreFlags: 0
vmwSTSGroupBaseDN: DC=vsphere,DC=local
vmwSTSUserBaseDN: DC=vsphere,DC=local
vmwSTSConnectionStrings: ldap://localhost:389
vmwSTSServiceUseMachineAccount: false
vmwSTSUserName: [email protected]
vmwSTSTimeout: 0
vmwSTSAuthenticationType: SRP
vmwSTSProviderType: IDENTITY_STORE_TYPE_VMWARE_DIRECTORY
vmwSTSDomainName: vsphere.local
vmwSTSDomainType: SYSTEM_DOMAIN

2. Get the FQDN of a linked vCenter which is still online.

3. Create an LDIF file to redirect vmwSTSConnectionStrings to that vCenter by entering the following commands.  Replace the dn: value with the LDAP distinguished name for the system domain configuration entry created in Preparation Step 3. 

echo 'dn: cn=vsphere.local,cn=IdentityProviders,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local' > fix.ldif
echo 'changetype: modify' >> fix.ldif
echo 'replace: vmwSTSConnectionStrings' >> fix.ldif
printf 'vmwSTSConnectionStrings: ldap://still-running-vcenter.example.com:389' >> fix.ldif

4. Repair the system domain by entering the following command, replacing the -D parameter with the Administrator account LDAP distinguished name created in Preparation Step 2.  Provide the password when prompted.

ldapmodify -h localhost -D "cn=Administrator, cn=Users, dc=vsphere, dc=local" -W -f fix.ldif

5. Verify the repair by re-running the command from Preparation Step 4 and making sure vmwSTSConnectionStrings value uses the hostname determined in step 2.

6. Restart the Gateway, then login to the GUI.

Additional Information

https://www.rfc-editor.org/rfc/rfc4514