VMware Cloud Foundation SDDC Manager Recovery Scripts

Products

• VMware Cloud Foundation

Issue/Introduction

As part of recovering from a failure in a VMware Cloud Foundation system, one of the following actions was performed:

• The NSX-V manager VM was redeployed.
• One, two, or three nodes of a NSX-T manager cluster was redeployed.
• A vCenter Server was redeployed using a file-based backup.
• The VxRail Manager VM was redeployed from a backup on a VMware Cloud Foundation on VxRail environment.
• The Management Domain was rebuilt, a Workload Domain was rebuilt, or a ESXi host was replaced without decommissioning it.

If one of the above action were performed, the following outcomes behaviour may occur.

• The SDDC Manager dashboard is missing information.
• A SDDC Manager workflow fails because SDDC Manager cannot communicate with a host or application.
• The host selection wizard either is not showing a warning/error it should be showing, the warning/error it is showing is incorrect, or the info icon reports incorrect information on a host.

Environment

VMware Cloud Foundation 3.x
VMware Cloud Foundation 4.x
VMware Cloud Foundation 5.x
VMware Cloud Foundation 9.x

Resolution

There are two procedures that resolve the above noted outcomes. One or both should be run, depending on the action you took.

• If you replaced an appliance or a host, follow the Refresh SDDC Manager SSH Keys procedure.
• If you replaced a host, in addition, follow the the Refresh SDDC Manager Host Attributes procedure. 

Note: These procedures are supported for VMware Cloud Foundation 3.8 and 4.x.

Refresh SDDC Manager SSH Keys

1. Download the correct script zip file for the VMware Cloud Foundation version currently deployed. See Script Versions note below for more information.
2. Copy the zip file into the SDDC Manager virtual appliance by using a secure file copy utility. Ensure to use the vcf account to authenticate and upload..
3. SSH to the SDDC Manager VM as the vcf user and switch to root:

su -

4. Create a temporary directory, such as /tmp/scripts, and unzip the files into it
5. run the following command. It validates the all SSH Keys that are there in the SDDC Manager and, if there are any discrepancies, asks the user to approve updating SDDC Manager's store.

Versions 5.x or lower: python refreshsshkeys.pyc

Versions 9.x or higher python refreshsshkeys.py

Refresh Host Attributes

1. If not already completed, do steps 1 to 4 in the above section. 
2. Use one of the following command variants to update the attributes of specific hosts

• To refresh Host Attributes for all hosts in the system: python refreshhostattributes.pyc --all
• To refresh Host Attributes of hosts that are part of a give domain: python refreshhostattributes.pyc --domain=<domain-name>
• To refresh Host Attributes of a list of given hosts: python refreshhostattributes.pyc --hosts=<comma-seperated-host-FQDNs>

Note: The host FQDN and domain names are reported in the SDDC Manager UI. You can also retrieve them using the SDDC Manager /v1/hosts and /v1/domains APIs. SDDC Manager APIs are documented at VMware {code} Community 

.

Note about Script Versions

The zip files contain compiled python scripts. You must run the python script that is compatible with your VCF version.

• VMware Cloud Foundation versions 3.8.0 to 3.9.0 inclusive should use the zip file recovery_tools_python_2_7_15.zip.

• VMware Cloud Foundation versions 3.9.1 to 5.0 inclusive should use the zip file recovery_tools_python_3_7_5.zip.

• VMware Cloud Foundation versions 5.1 to latest should use the zip file recovery_tools_python_3_10_11.zip.

• VMware Cloud Foundation versions 9.x should use the refreshsshkeys.py

Additional Information

Note: This script can be used on newer versions of VCF as well.