Disabling Port Group Reconfigure During Cell Site Host Provisioning

Products

VMware Telco Cloud Automation

Issue/Introduction

There is an issue which causes a port group reconfigure action to be triggered when cell site hosts are picked up for provisioning in a domain. When this reconfigure action is triggered, if there are any disconnected or unreachable hosts attached to the VDS then the reconfigure fails, thereby failing the provisioning of the hosts that were picked up and they are marked as FAILED.

This issue mainly happens if any of the port groups have vmnics that are marked to be set as unused uplinks.

Error:

Failed to reconfigure the vds: hostname reason: (vim.fault.DvsOperationBulkFault) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'Cannot complete a vSphere Distributed Switch operation for one or more host members.',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   hostFault = (vim.fault.DvsOperationBulkFault.FaultOnHost)

Environment

2.3

Resolution

Resolved in TCA 2.3.0.2 and 3.1.

Workaround:

This issue can be worked around using one of the two procedures listed in this article. Both procedures have been documented below so select the one that is more appropriate for your environment.

Procedure 1:

If the unreachable or disconnected hosts are removed from the relevant VDS containing the port groups that have unused uplinks, then provisioning of new hosts in this cell site group will succeed and any hosts previously failed due to this reason can be provisioned successfully after a resync.

Procedure 2:

The second workaround involves disabling the trigger to the port group reconfigure action i.e., disable functionality to allow changing of VLAN ID, Mac Learning Enablement, Uplink Port Order after the host provisioning has been done for a given VDS and port group combination in a cell site group.

However, because the functionality is being disabled, it means that even if a user changes the configuration in the cell site group and resyncs any existing hosts or adds new hosts, the port group configuration change is not performed. Therefore, any of these types of changes must be done manually in the relevant vCenter.

Ensure that there are no active workflows or workloads running on TCA and in the Infrastructure Automation component.
Take a snapshot of the TCA virtual machine for backup purposes.
SSH into the TCA Manager appliance.
Enter the Infrastructure Automation docker container with the command:
docker exec -it tcf-manager bash

Make a backup of the file named vcapi.py with the following command:
cp vcapi.py vcapi.py_bak

Open the file named vcapi.py for editing with VIM and navigate to line number 1135. The cursor should now be inside a function with the name create_or_modify_portgroup and it should look like the following code snippet before modification.

def create_or_modify_portgroup(
        self, pg_name: str, vlan_id: int, mac_learning_enabled: bool,
        uplink_port_order: UplinkPortOrder) -> Optional[vim.dvs.DistributedVirtualPortgroup]:
    """
    This method creates the specified port group if it is not present.
    If it is already present, then it is modified according to the specified configuration.
    @param pg_name: The name of the port group to be created/modified.
    @param vlan_id: The value for VLAN. If 0 is specified, then it means no VLAN.
    @param mac_learning_enabled: Whether to enabled mac learning or not.
    @param uplink_port_order: The grouping and ordering of the uplinks.
    @return The process port group
    @rtype Optional[vim.dvs.DistributedVirtualPortgroup]
    """
    pg = self.get_portgroup(pg_name)
    if pg:
        # PG already exists, so check if its current configuration is same as desired configuration
        existing_pg_spec = self.__compute_comparison_spec_from_pg(pg)
        desired_pg_spec = self.__compute_comparison_spec(vlan_id, mac_learning_enabled, uplink_port_order)
        # Complex Named Tuples can still be directly compared for equality
        if existing_pg_spec == desired_pg_spec:
            # Current config and desired config are same, so return existing PG object/instance
            return pg
 
        # Current and desired config are different, so reconfigure the PG
        new_spec = self.__compute_reconfigure_portgroup_spec(
            pg_name, pg.config.configVersion, vlan_id, mac_learning_enabled, uplink_port_order)
        self.__reconfigure_portgroup(pg, new_spec)
    else:
        # PG doesn't exist, so create it
        pg_spec = self.create_port_group_spec(pg_name, vlan_id, mac_learning_enabled, uplink_port_order)
        self.add_portgroup(pg_spec)
 
    return self.get_portgroup(pg_name)

Inside the if condition to check for the port group, add a return statement to return the port group immediately: return pg
The code snippet should look like this after this modification:

def create_or_modify_portgroup(
        self, pg_name: str, vlan_id: int, mac_learning_enabled: bool,
        uplink_port_order: UplinkPortOrder) -> Optional[vim.dvs.DistributedVirtualPortgroup]:
    """
    This method creates the specified port group if it is not present.
    If it is already present, then it is modified according to the specified configuration.
    @param pg_name: The name of the port group to be created/modified.
    @param vlan_id: The value for VLAN. If 0 is specified, then it means no VLAN.
    @param mac_learning_enabled: Whether to enabled mac learning or not.
    @param uplink_port_order: The grouping and ordering of the uplinks.
    @return The process port group
    @rtype Optional[vim.dvs.DistributedVirtualPortgroup]
    """
    pg = self.get_portgroup(pg_name)
    if pg:
        return pg
        # PG already exists, so check if its current configuration is same as desired configuration
        existing_pg_spec = self.__compute_comparison_spec_from_pg(pg)
        desired_pg_spec = self.__compute_comparison_spec(vlan_id, mac_learning_enabled, uplink_port_order)
        # Complex Named Tuples can still be directly compared for equality
        if existing_pg_spec == desired_pg_spec:
            # Current config and desired config are same, so return existing PG object/instance
            return pg

        # Current and desired config are different, so reconfigure the PG
        new_spec = self.__compute_reconfigure_portgroup_spec(
            pg_name, pg.config.configVersion, vlan_id, mac_learning_enabled, uplink_port_order)
        self.__reconfigure_portgroup(pg, new_spec)
    else:
        # PG doesn't exist, so create it
        pg_spec = self.create_port_group_spec(pg_name, vlan_id, mac_learning_enabled, uplink_port_order)
        self.add_portgroup(pg_spec)

    return self.get_portgroup(pg_name)
Note: Formatting is extremely important. Ensure the modified script looks exactly as it does in the example above.

After adding the return statement, the rest of the code inside the if statement block is unreachable, it can be left as-is or removed (optional).
Save the file and exit VIM
Exit the docker container with the ‘exit’ command.
Restart the docker container with the following commands:
systemctl stop tcf-managersystemctl restart tcf-manager-deploysystemctl start tcf-manager

After the above restart command returns control, list the running docker containers with the following command and ensure that the container named “tcf-manager’ is up and running:
docker ps -a

Any hosts added from this point forward can be provisioned successfully without being affected by this issue. Any hosts that failed previously due to this issue will now be provisioned successfully after a resync.

Additional Information

In the event a syntax error is introduced to the vcapi.py file, the tcf-manager container will fail to start.

In this scenario, the following command will open the appropriate file for corrections:

vi `docker inspect tcf-manager | grep MergedDir | cut -d "\"" -f 4 | sed 's/merged$/diff\/opt\/vmware\/tcf\/vcapi.py/'`

Follow step 11 above to restart the docker containers following any changes.