How to remove stale hosts from VCF inventory by enabling force to decommission

Products

• VMware Cloud Foundation
• VMware SDDC Manager / VCF Installer

Issue/Introduction

Symptoms

1. The Remove Selected Host option is grayed out in the SDDC Manager UI.
2. Hosts appear in an ERROR or STALE configuration state.
3. Inventory synchronization tasks fail to complete.

Environment

VMware Cloud Foundation 4.5.x and higher

Cause

Inventory mismatch occurs because out-of-band host removals do not trigger the necessary metadata cleanup in the SDDC Manager database.

Resolution

Currently, no resolution is available.

Workaround
There is already support (from VCF 4.5 release) in domain workflows (remove host from cluster) to remove of the stale hosts, if its removed from
unstandardized procedures out of VCF workflows.

From VCF 5.0 release onwards in decommission workflow, there will be a flag (default disabled) which needs to be passed to decommission workflow API in-case the Domain-workflow (remove host from cluster) fails to remove the stale host.

Steps:
This is supported from only API.

Note: Take a virtual machine snapshot of the SDDC Manager appliance in vCenter.

1. Either SSH to sddc-manager with vcf user and su to root or execute the API's from any rest client, provided the sample with curl command
   
   
2. Get the bearer auth token with following API
   
   TOKEN=$(curl -d '{"username" : "<sso_username>", "password" : "<sso_password>"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')
   
   
3. Copy the bearer token and provide same in bearer token section in the following decommission API by enabling force flag to remove only the stale hosts which are not required to be part of any workload domain, provide FQDNS of only dead hosts or stale hosts
   
   curl -k 'https://<sddc-manager.example.com>/v1/hosts?force=true' -i -X DELETE \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer $TOKEN' \
    -d '[{
           "fqdn" : "esxi-4.example.com"
         },
         {
           "fqdn" : "esxi-5.example.com"
         }
       ]'

4. Restart the SDDC Manager services to apply the changes:
   
   

   
   /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

 

Note: Please be cautious while providing the FQDN, provide only those FQDN's(stale hosts FQDN's) which are actually to be removed from VCF inventory, force flag will not do any check whether the host is part of any cluster or not, it will just cleanup inventory and host related configurations in SDDC-Manger.

Additional Information

For further assistance, refer to Contact Broadcom Support. For defects and enhancements, subscribe to this article for fix status updates.