Error "Service or view is not available at this time" and "The page can't be displayed" when accessing Fleet Lifecycle and accessing VCF Operations using VIDB

Products

• VCF Operations

Issue/Introduction

After patching VMware Cloud Foundation (VCF) Operations to version 9.1.0.x, the Fleet Lifecycle, Software Depot and vIDB (SSO) components become inaccessible. Users attempting to access the VCF Management interface may encounter the following symptoms:

• Error messages: "Service or view is not available at this time" or "503 Service Unavailable." 

• Component status: Within Manage > Fleet Management > Identity & Access > VCF SSO Overview, the Identity Broker shows below errors:

  • Identity Broker is not reachable

  • Identity Broker is unhealthy

• Login behavior: Users may be unable to log in to VCF Operations, receiving an error stating, "The page can't be displayed."

• In another instance: Access to the Lifecycle and Software Depot was restored for approximately 5 mins after rebooting the vcf_runtime VM. Analysis of /var/log/vmware/vpxd/vpxd.log indicates an IP conflict on management network between the VCF Control Plane VM and the VM having the same IP address as the VCF Control Plane VM

• • YYYY-MM-DDT05:43:45.405Z ERROR vpxd 309201 [vc@#### sub="InvtId" opID="HB-host-######@######-########"] Detected IP [dpg_Mgmt:10.##.#.###] conflict between vm [control_plane_vm_name] (moId:vm-######) nic:GuestInfo.net[0].ipAddress[0] and entity [duplicate_ip_address_vm_name] (moId:vm-######)

Environment

VCF Operations 9.1.x

VIDB 9.1.x

Fleet Lifecycle 9.1.x

Cause

The issue is caused by an IP address conflict between the active VCF Control Plane VM and a leftover, stale virtual machine (an Aria Operations for Networks instance but not limited to this specific component) that was not properly uninstalled from the environment or any other VM that holds the same IP address.

The presence of this stale/duplicate instance causes network routing and connectivity failures for the VCF Management/Fleet components. Additionally, the persistent registration of this instance in the lifecycle management inventory blocks subsequent upgrade pre-checks. The network communication failures ("No route to host" error confirmed from vCenter logs) due to the Control Plane IP (e.g., <control_plane_ip>) being incorrectly shared with the legacy instance (e.g., <stale_vm_name>).

Resolution

To resolve the IP conflict and restore accessibility to VCF Operations and Fleet Components, perform the following steps:

Prerequisites:

• Take a backup of the Fleet Components.

• Ensure that the IP addresses within the CIDR range are reserved exclusively for VMSP components (such as Fleet Lifecycle, SDDC Lifecycle, and Salt Master) and are not in use by any other appliances.

Steps:

1. Identify the Conflict: Analyze vCenter/Network logs to confirm that the IP address assigned to the active VCF Control Plane is also being utilized by a stale virtual machine in vCenter.

2. Remove Stale VM: In the vSphere Client, locate the legacy Aria Operations for Networks instance identified as <stale_vm_name>. Power off and delete the virtual machine to release the conflicting IP address. Note: Only proceed with powering off and deleting the VM after consulting and confirming with the appropriate stakeholders.

3. Change Network Configuration of VM using VCF Control Plane VM IP: 
   • Identify the conflicting VM and reconfigure its network adapter to use a free, non-conflicting IP address.
   • Verify that the IP address is exclusively assigned to the active VCF Control Plane VM .

   • Reboot the VCF Control Plane VM to clear the IP conflict state and re-establish service availability.

4. Clean Lifecycle Inventory: Navigate to the lifecycle management inventory from Build > Lifecycle > VCF Management > Upgrade. Identify and remove any lingering tasks, references, or entries associated with the deleted instance. Refer to Error: Undefined upgrade task in VCF 9.1 Operations lifecycle interface.

5. Validate Environment: Run the lifecycle upgrade pre-checks again to ensure they complete successfully.

6. Verify Accessibility: Navigate back to the VCF Management page to verify that the Fleet Manager and Identity Broker are accessible and no longer returning 503 errors.