Deploying the VCF Management Service failed - VCF services runtime cluster is missing from the inventory.

Products

• VCF Operations

Issue/Introduction

During upgrade attempt to VCF Operations 9.1, the following symptoms occur:

• VCF services runtime (VMSP) cluster is missing from inventory after VMs are manually deleted.
• Further upgrades from 9.0.2 to 9.1 are blocked.
• Fresh deployment fails because the workflow assumes an existing FLEET_LCM exists but uses an inconsistent API for verification.
• IP conflicts occur with addresses previously used by legacy components (Logs, Identity Broker, Network).

Environment

VCF Operations 9.0.2

VCF Operations 9.1

Cause

During a fresh VCF Operations 9.1 deployment, an IP conflict incident occurred - The VMs within the VSMP cluster were assigned IP addresses previously used by VCF Operations for logs, Identity Broker, and VCF Operations for network. vpxd logs confirm a user manually powered off and deleted all 6 VMSP VMs via the vSphere Client. However, the VMSP cluster record is now stale in SDDC Manager inventory; there is no customer-facing API to clean up those deleted VMs. This led to conflicting deployment workflow: 

Fresh new 9.1 deployment assumes that there is an existing FLEET_LCM and takes a different deployment path that doesn't involve deploying a new FLEET_LCM. It deploys a secondary-instance VCF services runtime (VMSP) cluster, then the workflow finds the existing FLEET_LCM and this fails as it uses a different API that doens't report an existing FLEET_LCM.

Resolution

Workaround:

Stale data needs to be cleaned up in the SDDC manager database and restart the workflow. No user-facing API is available for such clean up. Open a case through Broadcom Support Portal.