Certificate changes on one vCenter Server appear to affect another after Enhanced Linked Mode removal

Products

VMware vCenter Server

Issue/Introduction

After removing Enhanced Linked Mode (ELM) between two or more vCenter Server instances, they continue to behave as though they are still connected.
After replacing or renewing a certificate on one vCenter Server, certificates on a second vCenter Server stop being trusted at the same time, even though the two no longer replicate.

Certificates belonging to another vCenter Server appear in the trusted root store, similar to:

Alias : <sha1-fingerprint>
Entry type : Trusted Cert
Subject : CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=<other-vcenter-fqdn>, OU=VMware Engineering

Both vCenter Server instances report the same Single Sign-On domain identifier, even though each now appears to run its own domain.
One vCenter Server advertises far more plug-in or extension registrations than the number of objects it manages.
You see unresolved or grayed-out entries under Administration > Access Control > Global Permissions.
Service accounts belonging to another vCenter Server remain present, and they still hold administrative group membership.

Additional symptoms reported:

- I believe that the vCenters are somehow still linked.
- Certificate changes on one seems to affect the certificate on the other.
- I once enabled Enhanced Link Mode between 2 vCenters. I removed/disabled it because it was not what we needed.

Environment

VMware vCenter Server 7.0 and 8.0, where Enhanced Linked Mode was previously configured between two or more instances and later removed
Applies to embedded deployments in a shared Single Sign-On domain

Cause

This occurs when an Enhanced Linked Mode teardown removes directory replication but does not remove the objects the instances copied to each other while they were linked.

While vCenter Server instances are in Enhanced Linked Mode, they share a Single Sign-On domain and their VMware Directory Service (VMDir) databases replicate. Each instance therefore holds a full copy of the other's certificates, service accounts, permission entries and Lookup Service registrations.

When the link is removed, replication stops. The replicated objects do not. Each instance keeps a copy of the other's material with nothing to maintain or expire it.

The certificate symptom follows from this, and it is important to understand what it is not. The instances are not exchanging data, and nothing propagates between them. Each is holding a matching half of trust material minted while they were linked, in three places:

The peer's VMware Certificate Authority (VMCA) root in the trusted root store
On at least one instance, a Security Token Service (STS) signing certificate issued by the peer's VMCA
The peer's solution user certificate inside a Security Assertion Markup Language (SAML) relying-party registration

When you replace a certificate on one instance, the matching material still held by the other is invalidated at the same moment. To an administrator this presents as one instance affecting the other, when in fact both are losing validity against a shared past.

Two further conditions commonly reinforce the impression and are not themselves faults:

Both instances retain the Single Sign-On domain identifier from the period when they were linked. Removing an instance from a domain does not clear this value, so a shared identifier is expected and does not mean the instances are still connected.
The linkedvc: Linked Mode feature flag visible in the Managed Object Browser is a property of the vCenter Server Standard license edition. It does not indicate that Enhanced Linked Mode is configured.

Resolution

Perform steps 1 through 4 before making any change. Step 5 interrupts authentication on the instance where it runs and requires a maintenance window. Steps 6 through 9 do not require a service restart.

Confirm that replication is already severed on each instance. Connect over SSH and run:
```
/opt/likewise/bin/ldapsearch -h localhost -p 389 -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W -b "cn=Sites,cn=Configuration,dc=vsphere,dc=local" -s sub "(objectclass=vmwDirServer)" dn
```
Each instance must return only its own server object. If any instance returns another instance's server object, replication is still active and this article does not apply. Follow the standard Enhanced Linked Mode split procedure instead.
Identify the certificates belonging to the other instance. On each vCenter Server, run:
```
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text
```
Record the alias and subject of every entry whose subject names a vCenter Server other than the one you are logged in to. Leave your own VMware Certificate Authority root and any enterprise certificate authority roots in place.
Take a powered-off snapshot of every affected vCenter Server, without memory, and confirm that each snapshot restores. This is the only rollback for the remaining steps.
Download vCert from vCert - Scripted vCenter expired certificate replacement, upload it to each vCenter Server, and generate a certificate report from the main menu. Keep the report as your baseline.
Replace the Security Token Service signing certificate on any instance whose signing certificate was issued by another instance's VMware Certificate Authority. In vCert, select Manage certificates, then STS signing certificates. Restart services, then confirm that you can sign in as [email protected] and as a domain account before continuing.
Complete this step before step 6. The signing certificate is issued by a root that step 6 removes, and removing the root first leaves the token signing chain referencing a certificate that is no longer published.
Remove the other instance's VMware Certificate Authority roots from both stores on each vCenter Server. In vCert, select Manage certificates, then CA certificates in VMware Directory, then repeat using CA certificates in VECS Directory. Confirm each subject before you remove it. Restart services when the removals are complete.
Remove the Security Assertion Markup Language relying-party registrations that name the other instance:
```
/opt/likewise/bin/ldapdelete -h localhost -p 389 -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W -r "cn=<relying-party-url>,cn=RelyingParties,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local"
```
Enumerate the container rather than searching it for the other instance's host name. Registrations are commonly present under both the host name and the IP address, and a search by name misses the address entries.
Remove the other instance's solution users. These carry the other instance's identifier in their names and typically include hvc, wcp and workload_storage_management:
```
/usr/lib/vmware-vmafd/bin/dir-cli service delete --name <account-name> --login [email protected]
```
Deleting the service principal removes its group memberships at the same time, so no separate action is needed for those. Also remove any urn:acl:global:permissions entries under cn=AclModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local that name accounts belonging to the other instance.
Remove the other instance's Lookup Service registrations and topology objects. List the registrations, select those whose owner is a solution user of the other instance, and unregister each one:
```
/usr/lib/vmware-lookupsvc/tools/lstool.py unregister --url http://localhost:7090/lookupservice/sdk --id <service-id> --user [email protected] --no-check-cert
```
Then remove the other instance's deployment entry under cn=Ldus,cn=ComponentManager,dc=vsphere,dc=local and its site container under cn=Sites,cn=Configuration,dc=vsphere,dc=local. Remove the registrations before the site container, because the site container holds them and a recursive deletion removes them without the Lookup Service being aware.
Verify the result. Read the trusted root store directly with vecs-cli entry list --store TRUSTED_ROOTS rather than relying on the directory, confirm the registration count with lstool.py list, and confirm that Global Permissions shows no unresolved entries.

If you require the instances to occupy separate Single Sign-On domains:

The steps above leave both instances in their existing domain with the same domain identifier. This is not a functional link, but it does need addressing before an instance can join a different Enhanced Linked Mode configuration or use Hybrid Linked Mode.

Use cmsso-util domain-repoint to move the instance into a newly created Single Sign-On domain. This rewrites the identity source, the DNS zone, the Security Token Service credentials and the domain identifier in one supported operation.
Note that this carries a substantially larger change footprint and rollback cost than the targeted removal above. Select it only where separate domains are a stated requirement.

Additional Information

vCert - Scripted vCenter expired certificate replacement describes the certificate management tool used in steps 4 through 6.
Using the 'lsdoctor' Tool provides a read-only Lookup Service check. Running lsdoctor.py -l reports registrations grouped by Single Sign-On site, which shows the other instance's site and its contents clearly. Do not use the -u or -r options for this issue: -u recreates the local instance's solution users rather than removing another instance's, and -r rebuilds the local instance's registrations and does not affect the other instance's.
Procedure to split Enhanced Linked Mode (ELM) documents the intended separation procedure. Use that article when replication is still active. This article covers the objects left behind when a separation completes only partially.
Removing stale vCenter Server registrations from Enhanced Linked Mode covers cleanup after an instance is decommissioned without being unjoined. This article differs in that both instances remain in service.
The Single Sign-On domain identifier is not cleared when an instance leaves a domain. Two instances reporting the same identifier is expected after an Enhanced Linked Mode removal and is not evidence of a remaining connection.