Fixing VMDIR inconsistencies with SSO domain repoints

Products

• VMware vCenter Server

Issue/Introduction

The purpose of this article is to assist in repointing multiple vCenter Servers to a new SSO domain due to major inconsistencies within vmdir.   It uses an example of 3 vCenter Servers that are repointed from their current SSO domain to join a brand new single SSO domain.    It recreates the entire vmdir structure, which means all data that is stored in vmdir will be lost and recreated. There are some things that will need to be documented and recreated manually afterwards. This includes Global permissions, custom local SSO users and groups, and any external identity source(s) configured.  No UUIDs, certificates, roles, inventory permissions, VM parameters, performance data, are changed or lost. Because it recreates the SSO domain from scratch, it is the most guaranteed way to correct vmdir inconsistencies in an ELM environment.  The example below can be adjust though for the number of vCenter Servers being moved.  Add / Remove portions related to the "node" accordingly.

 

Environment

vCenter Server (VCSA) 6.7
vCenter Server (VCSA) 7.x
vCenter Server (VCSA) 8.x

Resolution

When using this guide, please ensure to update the FQDNs of the target nodes accordingly. Failure to do so may result in process failures and further inconsistencies.  

 

Sample vCenters:

 

node_a.vmware.com

node_b.vmware.com

node_c.vmware.com

 

 

Sample SSO domain name: vsphere.local

Friendly:  vsphere.local

DN:  dc=vsphere,dc=local

 

 

Prep work:

 

1.  Screenshot the AD identity source configuration (must be recreated after repoint of all nodes)

 

2.  Record all custom vsphere.local user and group information, like usernames and passwords (must be recreated after repoint of all nodes)

 

3.  Screenshot/record all global permissions (administation -> Global Permissions).  

 

4.  Record LDU-GUID of each vCenter server:

-- on each vCenter server, run the following command from command line and record the value:

/usr/lib/vmware-vmafd/bin/vmafd-cli get-ldu --server-name localhost

 

5.  Perform safe snapshots:

-- Record the ESXi host on which each vCenter server VM resides

-- Set the cluster DRS mode to manual (if applicable)

-- Login to the UI of each ESXi host and perform a guest OS shutdown on each vCenter server

-- Once all nodes are shut down, snapshot each VM.

-- Once each VM has been snapshotted, power them on again

 

NOTE:  If you must revert any node to snapshot, all nodes *must* be reverted at the same time.  Failure to do so will result in further replication problems. See KB 313886 for details.

 

 

Beginning with the chosen first node, we follow the repoint process for each node from command line.

6. Repoint the first node to a new SSO domain (node_a.vmware.com):

 

cmsso-util domain-repoint -m execute --src-emb-admin Administrator --dest-domain-name vsphere.local

 

7. Repoint the next node to the new SSO domain using the first node as the target (node_b.vmware.com):

 

cmsso-util domain-repoint -m pre-check --src-emb-admin Administrator --replication-partner-fqdn node_a.vmware.com --replication-partner-admin administrator --dest-domain-name vsphere.local

cmsso-util domain-repoint -m execute --src-emb-admin Administrator --replication-partner-fqdn node_a.vmware.com --replication-partner-admin administrator --dest-domain-name vsphere.local

 

8. Continue to repoint the nodes as needed using the previous node as the target (node_c.vmware.com):

 

cmsso-util domain-repoint -m pre-check --src-emb-admin Administrator --replication-partner-fqdn node_b.vmware.com --replication-partner-admin administrator --dest-domain-name vsphere.local

cmsso-util domain-repoint -m execute --src-emb-admin Administrator --replication-partner-fqdn node_b.vmware.com --replication-partner-admin administrator --dest-domain-name vsphere.local

 

Should an additional node require the same action, repeat the process again for the next node using the previous node as the target.

 

9. Upon moving the final node, an additional replication agreement should be created between the first node moved and the last node moved.  In our example, we will create a new agreement between "node_a" and "node_c" and will run the command from "node_c":

 

/usr/lib/vmware-vmdir/bin/vdcrepadmin -f createagreement -2 -h node_c.vmware.com -H node_a.vmware.com -u administrator

 

10. Run the following command on any one node to fix the internal Administrators group membership (copy all from /opt... to the last 'EOF' and paste into prompt - steps based upon Internal KB 314648).  If an additional node was repointed (more than what is provided in this KB), please add an additional "dn" entry for the targeted VCSA node:

 

----- Copy all below this line -----

 

/opt/likewise/bin/ldapmodify -x -D cn=Administrator,cn=Users,dc=vsphere,dc=local -W <<EOF

dn: CN=SystemConfiguration.Administrators,dc=vsphere,dc=local

changetype: modify

add: member

member: cn=Administrators,cn=Builtin,dc=vsphere,dc=local

 

dn: CN=ComponentManager.Administrators,dc=vsphere,dc=local

changetype: modify

add: member

member: cn=Administrator,cn=Users,dc=vsphere,dc=local

 

dn: cn=node_a.vmware.com,ou=Domain Controllers,dc=vsphere,dc=local

changetype: modify

replace: vmwLDUGuid

vmwLDUGuid: LDU-GUID-from-step-4

 

dn: cn=node_b.vmware.com,ou=Domain Controllers,dc=vsphere,dc=local

changetype: modify

replace: vmwLDUGuid

vmwLDUGuid: LDU-GUID-from-step-4

 

dn: cn=node_c.vmware.com,ou=Domain Controllers,dc=vsphere,dc=local

changetype: modify

replace: vmwLDUGuid

vmwLDUGuid: LDU-GUID-from-step-4

EOF

 

----- Copy all above this line -----

 

Only once all vCenter's are moved to the new SSO domain, and the group membership / LDU-GUIDs are fixed, should you move on to the final steps.  

 

11. Using the [email protected] account, login to the new ELM group and complete the following: 

 

12.  Recreate AD identity source as seen in step 1.  If the ID source was IWA, rejoin the AD domain accordingly. 

 

13.  Recreate local SSO users/groups as seen in step 2.

 

14.  Recreate global permissions as seen in step 3.

 

15.  On each vCenter server, recreate rbd waiter account.  For convenience we provide a script attached to KB 70635 - ESXI bootup through Auto-Deploy stuck at /vmw/rbd/host/xxxxxxxxxx/waiter.tgz - https://knowledge.broadcom.com/external/article?legacyId=70635

 

16.  Re-register any 2nd / 3rd party products that may have been previously registered with SSO (SRM/VR/NSX-V/etc.)

Additional Information

Repoint a vCenter Server Node to a New Domain -  https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vcenter.install.doc/GUID-9CB77874-D032-4C94-99AA-5340CB922F57.html
Repoint a vCenter Server Node to an Existing Domain with a Replication Partner - https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vcenter.install.doc/GUID-575C9B57-216B-4435-B410-D5AF1CECC026.html
Determining replication agreements and status with the Platform Services Controller (PSC) -  (https://kb.vmware.com/s/article/2127057#createagreement_parameter)
ESXI bootup through Auto-Deploy stuck at /vmw/rbd/host/xxxxxxxxxx/waiter.tgz - https://knowledge.broadcom.com/external/article?legacyId=70635
VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice - https://knowledge.broadcom.com/external/article/313886/vmware-vcenter-in-enhanced-linked-mode-p.html