SDDC Manager upgrade to 9.1 hangs at 'Installing VMSP services' when VCHA is enabled

Products

VCF Operations VMware vCenter Server

Issue/Introduction

An upgrade of SDDC Manager from version 9.0.x to 9.1 hangs indefinitely during the Installing VMSP services stage.

Reviewing the domainmanager.logs reveals that the VSP_BOOTSTRAP failed

[c.v.e.s.o.model.error.ErrorFactory,dm-exec-414] [KT3D6] PUBLIC_VSP_CLUSTER_BOOTSTRAP_FAILED Failed to bootstrap VCF services runtime
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to bootstrap VCF services runtime
...
...
Caused by: java.lang.NullPointerException: null

Technical analysis of the BootstrapVspClusterAction on the vCenter VM confirms the workflow is querying the vCenter inventory for the Management VM using its Fully Qualified Domain Name (FQDN).

DEBUG [vcf_dm,[GUID],d778] [c.v.v.q.i.p.vim.VimDataProvider,dm-exec-263] Sends the following request to '[VC_FQDN]': {"properties":["name","summary/runtime/powerState","guest/hostName","guest/ipAddress","guest/ipStack[*]/ipRouteConfig","guest/net[*]/ipConfig","@modelKey"],"resourceModel":"VirtualMachine","filter":{"criteria":[{"property":"guest/hostName","operator":"Like","comparableValue":"*[VC_FQDN]*","ignoreCase":true}],"operator":"And"},"sortCriteria":[{"property":"@modelKey","sortDirection":"Ascending"}],"offset":0,"limit":50,"returnTotalCount":true}
2026-05-13T19:19:24.383+0000 DEBUG [vcf_dm,[GUID],d778] [c.v.v.q.i.p.vim.VimResultConverter,vcf-ds-44] Received a response from '[VC_FQDN]'
2026-05-13T19:19:24.386+0000 DEBUG [vcf_dm,[GUID],d778] [c.v.evo.sddc.common.util.IPv4Utils,dm-exec-263] Calculated Network cidr 172.20.32.0/22
2026-05-13T19:19:24.388+0000 DEBUG [vcf_dm,[GUID],d778] [c.v.evo.sddc.common.util.IPv4Utils,dm-exec-263] Calculated Network cidr 172.20.32.0/22
2026-05-13T19:19:24.388+0000 DEBUG [vcf_dm,[GUID],d778] [c.v.evo.sddc.common.util.IPv4Utils,dm-exec-263] Calculated Network cidr 172.20.0.0/22
2026-05-13T19:19:24.389+0000 DEBUG [vcf_dm,[GUID],d778] [c.v.v.v.f.a.BootstrapVspClusterAction,dm-exec-263] The management network info {"ipAddress":"172.20.32.110","subnet":"172.20.32.0/22"}

As you can see from the last line, the gateway field is missing from the VmInfo.NetworkInfo object.

This is how this log line looks in other environments which were successfully deployed:

The management network info {"ipAddress":"########","gateway":"#######","subnet":"#######/23"}

Environment

VCF Operations 9.1.0

Cause

This issue occurs because the environment has vCenter High Availability (VCHA) enabled, resulting in multiple virtual machines (active and passive nodes) sharing the same FQDN.

The BootstrapVspClusterAction logic selects the first VM entry returned by the vSphere inventory search for a given FQDN. If the VCHA passive node—which is a clone of the active node—has a lower Managed Object Reference (MoRef) ID, it is incorrectly targeted for configuration.

Because the passive node does not maintain active network routing on its replication interface and lacks critical gateway information on its secondary interface, the VCF Services Platform (VSP) bootstrap process cannot communicate with required services, causing the task to hang.

Resolution

To resolve this issue, you must remove the duplicate FQDN entry by disabling VCHA before retrying the upgrade.

Destroy the vCenter High Availability (VCHA) configuration to ensure only a single virtual machine entry exists for the vCenter FQDN in the inventory.
Log in to VCF Operations.
Navigate to Build > Tasks > VCF Instances.
Locate the failed Deploy VCF Management Components task.
Click Restart Task.

Removing the VCHA configuration ensures the SDDC Manager upgrade workflow correctly identifies and connects to the active vCenter VM, which has the necessary network configuration to complete the VMSP service installation and subsequent component deployments (including ViDB).

Additional Information

Clusters that use vCenter High Availability (VCHA) are not supported in VCF 9.1, as documented in Supported and Not Supported Configurations to Converge to VCF.