The domain sync or the domain creation failure due to error in creating or updating Transit Gateway(TGW) on NSX manager
search cancel

The domain sync or the domain creation failure due to error in creating or updating Transit Gateway(TGW) on NSX manager

book

Article ID: 452453

calendar_today

Updated On:

Products

VMware Cloud Foundation VMware SDDC Manager / VCF Installer VMware NSX

Issue/Introduction

Domain sync:

After adding a new cluster to the VCF managed vCenter, the domain sync workflow fails at the 'Update or create with given network span' step with error
"InvalidRequest: Transit Gateway path=[/orgs/default/projects/default/transit-gateways/default] network stack cannot be changed from FULL_STACK_VPC to VLAN_BACKED_VPC."

Domain creation:

  • The domain is configured for NSX dedicated TEP overlay
  • The default transit gateway is associated with the default network span
  • The VPC connectivity profile associated with the default transit gateway is not associated with any private TGW IP blocks
  • There exists atleast one Virtual Private Cloud(VPC) created using default network span
  • When VLAN backed VPC domain is tried to create by sharing NSX manager from above mentioned domain configuration, then new domain creation fails with an error mentioning - "Error in creating or updating TGW on NSX manager for domain "



Environment

SDDC Manager 9.1.1
vCenter 9.1.1 or 9.1.0
NSX manager 9.1.0 or 9.1.1

Cause

Domain sync:

During the domain sync workflow with a newly added vCenter cluster, a custom network span is created to configure a VLAN-backed VPC. The workflow then attempts to associate this custom span with the default transit gateway in NSX Manager.
However, because the default transit gateway is already linked to the default FULL_STACK_VPC span, NSX Manager blocks the switch to the custom VLAN_BACKED_VPC span. This triggers the error:
"TGW /orgs/default/projects/default/transit-gateways/default has attached VPCs: {vpc list} consuming FULL_STACK_VPC feature sets. Changing network stack to VLAN_BACKED_VPC will impact existing VPCs."
Ultimately, this conflict causes the SDDC Manager domain sync workflow to fail.

Domain creation:

During domain creation, if there exists Virtual Private Cloud(VPC) on default network span which is also associated with default transit gateway(TGW). Then if a new VLAN backed VPC domain creation is issued by sharing that NSX manager, then the workflow tries to associate the VLAN backed VPC custom span with the default transit gateway. Since NSX does not allow changing transit gateway network stack from FULLSTACK VPC to the VLAN backed VPC in the presence of VPC, create domain workflow fails with below error.

Resolution

Domain sync:

  1. Freshly re-trigger the domain sync from SDDC manager for the sync failed domain.
  2. Retrieve the custom VLAN Backed VPC span that is associated with the newly synced cluster from the NSX manager.
  3. Create a new transit gateway in the NSX manager and associate it with the span retrieved from step 2

Domain creation:

  1.  Get the network span name associated with the VLAN backed VPC cluster of the new domain from vCenter from the screen as below. Focus on cluster Cluster, then navigate to Configure->Hosts->Networking-> Network Configuration and check for the Network span name.

  2. Identify the failed create domain task from the SDDC manager rest-api 

    GET https:///api/v1/tasks Get the value of "id" field for the failed create domain workflow from the above api response

  3. Trigger the rest-api to get the workflow spec by providing ID retrived from step 2 and store in a temporary location 

       curl -s -X GET http:///domainmanager/internal/vault/{id} | json_pp > /tmp/workflow.json 

  4.  Edit the /tmp/workflow.json file 

       4a. Set the field "suggestedIdForNewTgw" to the value "tgw-for-span-<network-span-id>" 
           where <network-span-id> is the ID of the network span retrieved from step 1 

       4b. Set the field "useDefaultTgw" as false, as shown below and save the file. 
           "NsxAddViEngine____17__NsxtPolicyAddVi____17__UpdateOrCreateTgwWithNetworkSpan____11__suggestedIdForNewTgw" : "tgw-for-span-sddc-cde5d2d5-737b-42c7-9aeb-74438d4568e2", "NsxAddViEngine____17__NsxtPolicyAddVi____17__UpdateOrCreateTgwWithNetworkSpan____11__useDefaultTgw" : false,

  5. Trigger the rest-api to update the inputs for failed create domain workflow

       curl -s -X PUT -H "Content-type: text/plain" http:///domainmanager/internal/vault/{workflow-id} -d @/tmp/workflow.json

  6. Retrigger the failed create domain workflow from the SDDC manager UI.
Powered by Wolken Software