VMDir State Read-only with stale server in SSO domain

Products

• VMware vCenter Server

Issue/Introduction

Warnings: 

• This KB is not for use when you have servers configured with Enhanced Linked Mode.
• This KB is not for use to remove a vCenter/PSC that is being retired.  
• This KB is not for use to remove replication agreements between two vCenters.  
• This KB makes use of examples like "vcenter2.example.local", these are examples only.  Hostnames will be based on the environment.  No commands should be repeated with example hostnames in use.  

Symptoms: 

• vCenter is not linked to any other vCenter using Enhanced Linked Mode.  
• VMDir State continues to go to Read-only. 
  
  #/usr/lib/vmware-vmafd/bin/dir-cli state get
Read-only
  
  
• All services start successfully.
• Verify replication shows a second server that doesn't exist in the environment.  
  
  #cd /usr/lib/vmware-vmdir/bin
#./vdcrepadmin -f showservers -h localhost -u administrator

Example Output
cn=vcenter1.example.local,cn=Servers,cn=home,cn=Sites,cn=Configuration,dc=vsphere,dc=local
cn=vcenter2.example.local,cn=Servers,cn=home,cn=Sites,cn=Configuration,dc=vsphere,dc=local

• It should only list vcenter1.example.local.
• The commands showpartners and showpartnerstatus show nothing.
  
  #./vdcrepadmin -f showpartners -h localhost -u administrator
#./vdcrepadmin -f showpartnerstatus -h localhost -u administrator


• vmafdvmdirclient.log shows replication errors:

  YYYY-MM-DDTHH:MM:SS.###Z:t@############:ERROR: VmDirSafeLDAPBindEx to (ldap://<vCenter>:389) failed. SRP(9127)
YYYY-MM-DDTHH:MM:SS.###Z:t@############:ERROR: VmDirAnonymousLDAPBindEx to (ldap://<vCenter>:389) failed. (-1)(Can't contact LDAP server)
YYYY-MM-DDTHH:MM:SS.###Z:t@############:ERROR: _VmDirGetDSERootAttributeEx failed with error (9127)

Environment

vCenter 7.x

vCenter 8.x

Cause

Sometime in the past a vCenter or PSC was removed from the environment but it was not removed from vCenter.  

Resolution

Warning: Following this KB for the wrong vCenter can destroy the vCenter.  It is very important to verify the vCenter does not exist before following the steps below.

1. Verify vcenter2.example.local does not exist in the environment by searching vCenter by name.  PING the name and do an nslookup for the name.  Take any other steps to verify the vCenter doesn't exist before continuing with the steps.    
2. Take a powered off snapshot of the vCenter.  If the vCenter is linked to other vCenters using Enhanced Linked Mode then you must take a powered off snapshot of all linked vCenters at the same time.
3. SSH to the vCenter.
4. Set VMDir State to Normal.
   
   #/usr/lib/vmware-vmdir/bin/vdcadmintool
   

   Choose Option 5. Set vmdir state 

NORMAL

   Choose Option 0.  exit

5. Run the command to remove the stale vCenter/PSC entry.  
   

   #cd /usr/lib/vmware-vmdir/bin

   #cmsso-util unregister --node-pnid <vCenter_FQDN_that_will_be_removed> --username [email protected] --passwd 

   If the operation completes successfully it will say "Success".

6. Reboot vCenter and verify all services start.
7. Verify VMDir State is Normal.
   
   #/usr/lib/vmware-vmafd/bin/dir-cli state get 
Normal

Additional Information

Determining replication agreements and status with the Platform Services Controller (PSC)

Using the cmsso command to unregister vCenter with External PSC or vCenter with Embedded PSC from Single Sign-On