How to Remove Stale Datastores Left by HCX

Products

• VMware HCX
• VMware vCenter Server

Issue/Introduction

After recovering from a failed HCX Interconnect deployment using force redeployment or improper cleanup of the IX appliance, environments may contain a stale datastore(ma-ds-###########) that remains in maintenance mode and cannot be removed through standard vCenter operations. This situation occurs when complications arise during the redeployment process, though the exact circumstances leading to the datastore entering maintenance mode may vary.

Safe datastore removal requires verification of several conditions. The first step involves reviewing the HCX service mesh for any active migrations or ongoing tasks related to the Interconnect appliances. A failed deployment state typically manifests as an Interconnect with a "-Redeploying" suffix in its name or displays a "Failed" or "Error" status. Multiple versions of the same Interconnect appliance in vCenter indicate a previous deployment issue.

Verification of no active operations and identification of the failed deployment state must be completed before proceeding. The current Interconnect must be confirmed healthy and operational before removal of the stale datastore. This verification prevents service disruptions that could occur from removing datastores associated with an active Interconnect.

Environment

VMware vCenter Server 7.x and later
VMware HCX 4.x and later

Cause

This issue involves several contributing factors that can occur during the redeployment process:

1. An initial failed deployment of an HCX Interconnect that requires recovery action. This failure might occur due to various reasons such as datastore inaccessibility or other infrastructure issues.
2. A subsequent force redeployment successfully creates a new interconnect instance to recover from the failed state. While this resolves the immediate connectivity issues, a datastore from the previous deployment may remain in the environment.
3. During this sequence of events, a datastore may enter maintenance mode. The timing and trigger of the maintenance mode state can vary depending on environmental factors and administrative actions.
4. After the successful force redeployment creates new healthy datastores, references to the previous datastores can persist in the vCenter database, making standard removal methods ineffective. When persistent issues occur during the removal process, engaging VMware support is recommended for additional guidance.

Resolution

If you believe you have encountered this issue, please open a support case with Broadcom Support and refer to this KB article.
For more information, see Creating and managing Broadcom support cases