After restoring SDDC Manager from a backed-up state that was configured in Multi-Site, Federation operations may fail.

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• If one of the nodes in a Multi-Site Federation is restored from a backup (using VCF Back & Restore utility), it may not have the latest state of the Federation.
• Join/Leave operations will fail and "Failed to join the federation. Errors: Internal server error: Failed to download Certificate" error messages are seen in the Multi-Site Dashboard.
• The VMware Cloud Foundation instance health on the Multi-Site Dashboard is not Green.

Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.

Resolution

Once the the node is restored from a back-up state, as it may not have the latest Federation state, the recommendation is to evict (force leave) the SDDC Manager instance from the Federation and join the instance back if required.

This can be achieved using the Multi-Site Evict API.
Note: Evict should be executed from SDDC Manager which has role as 'Controller' in the Federation.
Below are the steps to execute the Evict API.

1. Login to SDDC Manager (which was assigned as Controller in the Federation) as user 'vcf'.
2. Execute the below API from get an Access Token to be used for later steps:

curl 'https://<SDDC-Manager-FQDN>/v1/tokens' -i -X POST \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/json' \
    -d '{
  "username" : <admin username>,
  "password" : <admin password>
}'

3. Execute the Evict API from SDDC Manager using the acquired Access Token.

curl 'https://<Controller-SDDC-Manager-FQDN>/v1/sddc-federation/members/<<Evicted-SDDC-Manager-FQDN>>?force=true' -i -X DELETE \
    -H 'Authorization: Bearer <access token>'

4. Navigate to the Multi-Site Dashoard and confirm the evicted instance is no longer visible. 

Note: For information on the VMware Cloud Foundation API, see the API documentation guide .

5. Add the restored instance back into the Federation from the Multi-Site Dashboard.
6. Confirm that all the existing instances in the Multi-Site Dashboard should be now healthy (Green).
7. Re-try any failed Join & Leave operations.