VCF 9.1 deployment fails to bootstrap VCF Services Runtime when a network name resolves to multiple vCenter network objects

Products

• VCF Operations

Issue/Introduction

During a VMware Cloud Foundation 9.1 deployment, the workflow fails at Deploy and configure VCF management platform while attempting to bootstrap the VCF Services Runtime.

The VCF Installer displays an error similar to:

• Failed to bootstrap VCF services runtime
• PUBLIC_VSP_CLUSTER_BOOTSTRAP_FAILED

The /var/log/vmware/vcf/domainmanager/domainmanager.log file contains an OVA import failure similar to:

• Error: failed to import
vcf-services-platform-template-9.1.0.0.<build>.ova:
path '/<datacenter>/network/Management Network' resolves to multiple networks

The VCF Services Platform OVA import failed because the network inventory path resolved to more than one network object.

Environment

VMware Cloud Foundation 9.x

Cause

The network selected for the VCF Services Runtime deployment did not have a unique display name within the vCenter Datacenter inventory.

Example: Multiple vCenter network objects were named:

• Management Network

These duplicate objects may include:

• Multiple distributed port groups with the same name
• A distributed port group and one or more standard port groups with the same name
• Network objects located in different network folders
• Stale or unused network objects retaining the same name

The deployment workflow initially identifies the selected distributed port group by its MoID, the VCF Services Platform OVA deployment utility also resolves the network using the vCenter inventory path.

Path resolved to multiple objects, the deployment utility could not uniquely identify which network should be assigned to the OVA:

• path '/sacny/network/Management Network' resolves to multiple networks

The OVA import failed before the temporary bootstrap VM was created. The later StringIndexOutOfBoundsException and bootstrap-VM-not-found messages were secondary errors generated when the workflow attempted to check the status of a bootstrap VM that did not exist.

Resolution

Identify all standard and distributed port groups using the duplicate network name and ensure that the distributed port group selected for VCF Management Services has a unique name.

 

1. Identify the selected network MoID from the VCF logs

On the VCF Installer or SDDC Manager, run:

grep -nEi \
'networkMoId|VSPHERE_NETWORK|resolves to multiple networks' \
/var/log/vmware/vcf/domainmanager/domainmanager.log

 

For rotated logs:

zgrep -nEi \
'networkMoId|VSPHERE_NETWORK|resolves to multiple networks' \
/var/log/vmware/vcf/domainmanager/domainmanager*.log.gz

Example:

• networkMoId: dvportgroup-248646

The reported deployment selected dvportgroup-248646, but the OVA import failed because the inventory path for Management Network resolved to multiple network objects.

 

2. Identify duplicate networks in the vSphere Client

Log in to the vSphere Client.

Select Menu > Networking.

1. Select the affected vCenter Server and Datacenter.
2. In the networking inventory, locate all network objects named:
   • Management Network
3. Expand each vSphere Distributed Switch and network folder to determine whether more than one distributed port group has the same name.
4. Select each matching distributed port group and record:
   • Distributed switch name
   • VLAN ID
   • Port binding type
   • Number of connected virtual machines
   • Number of connected hosts
5. Select Menu > Hosts and Clusters.
6. For each ESXi host in the affected Datacenter, select:
   • Host > Configure > Networking > Virtual switches
7. Check whether a standard port group named Management Network also exists on one or more ESXi hosts.

A naming overlap may exist between:

• Two distributed port groups
• A distributed port group and standard port groups
• Network objects in different network folders
• An active network and a stale or unused network object

 

3. Verify the distributed port group MoID in the vSphere Client

1. In Menu > Networking, select the distributed port group used for VCF Management Services.
2. Review the browser address bar.

The URL normally contains an object identifier similar to:

• DistributedVirtualPortgroup:dvportgroup-248646

or:

• urn:vmomi:DistributedVirtualPortgroup:dvportgroup-248646

Confirm that the MoID matches the value found in domainmanager.log.

Browser URL formats can vary between vSphere Client builds.

 

4. Rename the selected distributed port group in the vSphere Client

Where possible, rename the distributed port group selected for VCF Management Services rather than a standard port group carrying ESXi management traffic.

1. Select Menu > Networking.
2. Expand the affected vSphere Distributed Switch.
3. Right-click the distributed port group selected for VCF Management Services.
4. Select Rename.
5. Enter a unique name, for example:
   • vcf-management-services
6. Save the change.
7. Select the renamed distributed port group.
8. Verify that the following configuration remains unchanged:
   • Distributed switch
   • VLAN ID
   • Port binding
   • Teaming and failover policy
   • MTU
   • Security policy
   • Connected hosts
   • Connected virtual machines

*Do not rename a standard port group carrying the ESXi management VMkernel adapter unless the change has been planned and its effect on host management connectivity has been evaluated.

6. Verify that the MoID remains unchanged

After renaming the distributed port group, run:

• Get-View -Id "DistributedVirtualPortgroup-dvportgroup-248646" |
    Select-Object Name,
        @{N="MoID";E={$_.MoRef.Value}}

Expected result:

Name                     MoID
----                     ----
vcf-management-services  dvportgroup-248646

 

The display name should change while the managed object ID remains the same.

7. Confirm that the network name is now unique

In the vSphere Client:

1. Select Menu > Networking.
2. Search for the new distributed port group name.
3. Confirm that only one network object uses that name.
4. Confirm that the original Management Network name no longer produces an ambiguous match for the network intended for VCF Management Services.

Using PowerCLI:

Get-VDPortgroup |
    Group-Object Name |
    Where-Object {$_.Count -gt 1} |
    Select-Object Name, Count

 

Review any remaining duplicate distributed port group names that could affect the deployment.

 

8. Retry the failed VCF deployment task

Retry the failed Bootstrap VCF Services Platform task from the VCF Installer interface.

The failure occurred before the bootstrap VM was created, and the task was reported as retryable.

After retrying, confirm that domainmanager.log no longer contains:

• resolves to multiple networks

Verify that:

1. The VCF Services Platform OVA imports successfully.
2. A temporary bootstrap-vm-<generated-suffix> VM is created.
3. The workflow advances from:
   • Bootstrap VCF Services Platform

to:

• • Configure VCF Services Platform

Additional Information

https://knowledge.broadcom.com/external/article/441554/vcf-automation-or-vcf-identity-broker-in.html