Host Provisioning in TCA Infrastructure Automation Fails with "Index out of range" Message

Products

• VMware
• VMware Telco Cloud Automation

Issue/Introduction

Symptoms:
Provisioning an ESXi host in Telco Cloud Automation 2.1.x and earlier may fail with error message "Index out of range".

The /common/logs/ztp/tcf-manager.log will have an error message similar to below:

Exception('Failed to migrate networking for ESXi: examplehost001.domain.com,
and exception: list index out of range'), 'result_message': 'exception'} and
exception is: Failed to migrate networking for ESXi: examplehost001.domain.com,
and exception: list index out of range

Environment

VMware Telco Cloud Automation 2.0
VMware Telco Cloud Automation 2.0.1
VMware Telco Cloud Automation 1.x
VMware Telco Cloud Automation 2.1

Cause

This can occur when the target DVS uplink portgroup contains stale ports.

This can be verified by navigating to the uplink portgroup in vCenter and sorting by name. 

Stale uplink ports will not have values in the Name and Connectee field:

Resolution

Failed host provisioning via Infra Automation due to stale DVS ports will be addressed in a future TCA release. Stale ports can be removed manually with the below workaround to complete successful host provisionings.

Workaround:

1. Locate and record the stale port IDs from the vCenter uplink portgroup port list (stale uplink ports will not have values in the Name and Connectee field)
2. Take a backup of the DVS (https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-9B33EBCE-1BCC-4362-9BE5-4F477056B169.html )
3. Open the vCenter MOB in a browser and log in with administrator account [Ex. https://vcenter01.domain.com/mob]
4. Navigate to the MOB object for the DVS. This can be accessed by clicking "content" > rootFolder "group-d1" > child entity for the datacenter (ex. "datacenter-3) > networkFolder (ex. "group-n7") > childEntity (ex. "dvs-105")
5. Select action ReconfigureDVPort_Task at the bottom of the page.
6. Enter the following content (replace port ID for the stale port ), then click Invoke Method

<port>
   <operation>remove</operation>
   <key>4440</key>
</port>

Repeat for any additional stale ports.

The ESXi host with Failed provisioning status in TCA can then be removed and re-added to provision successfully.