Removal of stale VCF components from VCF Operations after decommissioning management domain (VCF Instance)

Products

• VCF Operations

Issue/Introduction

This article outlines the specific scenarios to clean stale components from VMware Cloud Foundation (VCF) Operations 9.1 deployment while decommissioning the management domain (VCF Instance).  Manual cleanup is required in the following instances:

1. Management domain (VCF instance) has completely decommissioned.

2. All component clean-up via script "cleanup_component.py" as per Failed component cleanup during VCF 9.1 installation or upgrade has already been performed.

3. All component VMs like  (SDDC, VMSP) has been deleted.

4. The VCF Operations Software Depot page still references the old Fleet Manager's hostname

5. Checking https://<ops_hostname>/casa/services shows old VCF services are active or listed.

6. Running internal API checks against https://<ops_hostname>/suite-api/internal/components returns multiple stale component types such as SALT_MASTER, SALT_RAAS, FDS, FLEET_LCM, SDDC_LCM, VSP.

   • Note : Some of the VCF components might not be present in the VCF instance that is being decommissioned.

Environment

• VMware Cloud Foundation 9.1
• VCF Operations 9.1

Cause

Manually deleting or decommissioning VCF management domain (VCF Instance) virtual machines bypasses the standard unregistration workflows, leaving stale, orphaned component UUID mappings inside the VCF Operations internal API database inventory.

Resolution

To completely remove the stale component remnants from VCF Operations component inventory, you must manually evict them utilizing internal API endpoints.

Prerequisites :

1. Place the VCF Operations cluster Offline from VCF Operations admin console.

2. Take an non-memory snapshot of the VCF Operations node(s).

3. Bring the cluster back online before executing the API calls below.

Procedure :

1. SSH to the VCF Operations node.

2. Acquire  API security token by sending a POST request to the authentication endpoint: 

   curl -k --request POST \
   
     --url https://<ops_hostname>/suite-api/api/auth/token/acquire \
   
     --header 'accept: application/json' \
   
     --header 'content-type: application/json' \
   
     --data '{
   
     "username": "admin",
   
     "password": "<PASSWORD>"
   
   }'

3. Fetch the full inventory list of active components from the internal API endpoint using the acquired token:

   1. Create a file named test.sh 

      1. vi test.sh

      2. Add the below content to the test.sh 

         #!/usr/bin/bash
         
         curl -k --request GET \
         
           --url https://<ops>/suite-api/internal/components \
         
           --header 'accept: application/json' \
         
           --header 'authorization: vRealizeOpsToken <token>' \
         
           --header 'content-type: application/json' \
         
           --header 'x-vrealizeops-api-use-unsupported: true'

      3. chmod +x test.sh

   2. Run : ./test.sh | python -mjson.tool > components.json

   3. Run : grep -E 'component(Type|Uuid)' components.json | grep -v "           " | sed -nre 's/\s+/ /p' > component.txt

4. Locate and note the componentUuid strings associated with the stale componentType entries (e.g. delete in the exact same order : SALT_MASTER, SALT_RAAS, FDS, FLEET_LCM, SDDC_LCM, VSP).

   • Note : Do not delete the OPS componentUuid, as this represents the core VCF Operations instance.

5. For each stale componentUuid identified in Step 3c, issue a DELETE request: 

   curl --request DELETE \
   
     --url https://<ops_hostname>/suite-api/internal/components/<uuid> \
   
     --header 'accept: application/json' \
   
     --header 'authorization: vRealizeOpsToken <token>' \
   
     --header 'content-type: application/json' \
   
     --header 'x-vrealizeops-api-use-unsupported: true' -k

   • Note: FLEET_LCM and VSP components may take longer than 5 minutes to clear out completely. Allow the background processes time to finish.

   • Note: If the deletion of the component fails due to an invalid or unreachable hostname, navigate to the Management Tasks page within the VCF Operations user interface, locate the failed task, and select the Skip Task checkbox to force completion.

6. Verify that the inventory has been successfully cleared by re-running the GET command from Step 3.a.ii. The output should now exclusively return the core OPS component.

7. Confirm that the Software Depot page within the VCF Operations interface has correctly reverted to its empty introductory state.