VMware NSX 4.x
The need for multiple CARR runs stems from the cross-site trust model used in NSX Federation.
Federation environments rely on Principal Identities (PIs) to trust and communicate securely between sites (GM <-> LM, GM <-> GM). When a site's Local or Global Manager Platform Certificate is changed, the corresponding Principal Identity on the remote site must be updated with the new certificate.
When CARR runs on a site, it first remediates all self-signed certificates local to that site. After the local certificate is replaced, it is automatically pushed via site sync to the other LM or GM sites to update the remote Principal Identity.
The CARR remediation process is not fully aware of the stale, expired certificates that were replaced on remote sites during previous operations. The synchronization of the newly pushed certificate does not automatically clean up the previously replaced (now stale) certificate entries on the remote node's trust store. This results in stale certificate entries that remain on the remote appliance with a "Used By" value of CLIENT_AUTH. These stale certificates are not local to the original site, and therefore, CARR cannot identify and delete them in the first local run.
In Federation environments CARR script must be run at least twice on all sites to complete self signed certificate rotation.
Stale certificates will be removed automatically on additional runs.
Note: CARR script should be run on all sites completely first before running again for a second time.