VCF 9.1 Workload Domain Sync Fails with "No transport zone found for given path"
search cancel

VCF 9.1 Workload Domain Sync Fails with "No transport zone found for given path"

book

Article ID: 454526

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer VMware NSX

Issue/Introduction

  • Workload domain synchronization in VCF Ops 9.1 using VCF Ops > WLD Domain > Actions > Sync Inventory fails during the "Fetch TNPs for successfully configured clusters" operation with the below error:

    No transport zone found for given path: /infra/sites/default/enforcement-points/default/transport-zones/<transport_zone_uuid>



  • The /var/log/vmware/vcf/domainmanager/domainmanager.log on the SDDC Manager shows "No transport zone found for given path" error similar to below for one or more <transport_zone_uuid> :

    YYYY-MM-DDThh:mm:ss DEBUG [vcf_dm,<dm_task_id>,<dm_subtask_id>] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-1217]  Invoking task FetchTransportNodeProfilesAction., Plugin: NsxtPlugin
    ...
    YYYY-MM-DDThh:mm:ss DEBUG [vcf_dm,<dm_task_id>,<dm_subtask_id>] [c.v.v.c.n.s.c.c.ApiConnection,dm-exec-1217]  Creating ApiClient to https://<NSX_MGR_FQDN>:443 with username svc-<NSX_svc_account>
    ...
    YYYY-MM-DDThh:mm:ss ERROR [vcf_dm,<dm_task_id>,<dm_subtask_id>] [c.v.v.c.f.p.n.a.FetchTransportNodeProfilesAction,dm-exec-1217]  Error fetching transport node profiles
    java.util.NoSuchElementException: No transport zone found for given path /infra/sites/default/enforcement-points/default/transport-zones/<transport_zone_uuid>
            at com.vmware.vcf.common.nsxt.sdk.client.connection.VcfNsxModelTranslationEnhancer.loadIdsForPolicyBasedStandardHostSwitch(VcfNsxModelTranslationEnhancer.java:250)
            ...
            at com.vmware.vcf.common.fsm.plugins.nsxt.action.FetchTransportNodeProfilesAction.execute(FetchTransportNodeProfilesAction.java:92)
    ...
    YYYY-MM-DDThh:mm:ss ERROR [vcf_dm,<dm_task_id>,<dm_subtask_id>] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-1217]  [<TOKEN>] NSX_TNC_CONFIG_ERROR_GENERIC Error in configuring transport node collections
    com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Error in configuring transport node collections
            at com.vmware.vcf.common.fsm.plugins.nsxt.action.FetchTransportNodeProfilesAction.execute(FetchTransportNodeProfilesAction.java:100)
            at com.vmware.vcf.common.fsm.plugins.nsxt.action.FetchTransportNodeProfilesAction.execute(FetchTransportNodeProfilesAction.java:34)
    ...
    Caused by: java.util.NoSuchElementException: No transport zone found for given path /infra/sites/default/enforcement-points/default/transport-zones/<transport_zone_uuid>
            at com.vmware.vcf.common.nsxt.sdk.client.connection.VcfNsxModelTranslationEnhancer.loadIdsForPolicyBasedStandardHostSwitch(VcfNsxModelTranslationEnhancer.java:250)
            ...
            at com.vmware.vcf.common.fsm.plugins.nsxt.action.FetchTransportNodeProfilesAction.execute(FetchTransportNodeProfilesAction.java:92)

  • Additionally the /var/log/vmware/vcf/operationsmanager/operationsmanager.log shows "NoSuchElementException" when attempting to query the concerned Transport zone similar to below:

    YYYY-MM-DDThh:mm:ss INFO  [vcf_om,<om_task_id>,<om_subtask_id>] [c.v.v.c.n.s.c.c.NsxtManagerTransportNodeOperationsPolicyHandler,pool-2-thread-9] nsx-policy-transportnode: Returning the stream of transport nodes using the policy API.
    ...
    YYYY-MM-DDThh:mm:ss ERROR [vcf_om,<om_task_id>,<om_subtask_id>] [c.v.v.b.p.e.b.AbstractEntityBuilder,pool-2-thread-9] Entity builder failed with exception java.util.NoSuchElementException: No transport zone found for given path /infra/sites/default/enforcement-points/default/transport-zones/<transport_zone_uuid>

  • Attempting to manually fetch the Transport Zone details for the affected <transport_zone_uuid> via LIST Transport zone API from the SDDC or NSX Manager confirms the affected Transport zone details are missing:

    curl -sk -u 'svc-<NSX_svc_account>' 'https://<NSX_MGR_FQDN>/policy/api/v1/infra/sites/default/enforcement-points/default/transport-zones?page_size=1000' > /tmp/tz-list.json && jq '{result_count, returned: (.results|length)}' /tmp/tz-list.json && jq -r '.results[].id' /tmp/tz-list.json | sort > /tmp/tz-ids.txt && grep -E '<transport_zone_uuid>' /tmp/tz-ids.txt || echo 'ABSENT FROM LIST'

  • However manual curl attempt from SDDC Manager with same NSX service account is able to fetch the concerned Transport Zone details:

    vcf@<SDDC> [ ~]$ curl -1 -k -u 'svc-<NSX_svc_account>' -H 'Content-Type:application/json' -X GET https://<NSX_MGR_FQDN>/policy/api/v1/infra/sites/default/enforcement-points/default/transport-zones/<transport_zone_uuid>

    Enter host password for user 'svc-<NSX_svc_account>' : (prompts for NSX Manager password to be typed in)
    {
    "nsx_id" : "####-####",
    "tz_type" : "<VLAN/OVERLAY>",
    ....
    "resource_type" : "PolicyTransportZone",
    "id" : "<transport_zone_uuid>",
    "display_name" : "<transport_zone_name>",
    "description" : "<description>",
    "path" : "/infra/sites/default/enforcement-points/default/transport-zonps/<transport_zone_uuid>",
    "relative_path" : "<transport_zone_uuid>",
    "parent_path" : "/infra/sites/default/enforcement-points/default",
    "remote_path" :
    "unique id" : "<transport_zone_uuid>",
    "realization_id" : "<transport_zone_uuid>",
    ...
    vcf@<SDDC> [ ~]$

Environment

VCF 9.1

NSX 9.1

Cause

The synchronization failure results from an inconsistency in the NSX Policy representations of the transport zone. The transport zone is visible in the NSX Manager configuration but is unavailable in the NSX Policy configuration accessed by SDDC Manager during Sync Inventory operation

Resolution

This is a known issue identified by Broadcom Engineering and will be fixed in an upcoming NSX release.

To workaround the issue, follow the below steps:

  1. Obtain the fix_missing_parent_path.py script from the attachments section of this article

  2. Copy the script to the target NSX Manager node (Local Manager in case NSX is Federated) using SCP:

    scp fix_missing_parent_path.py root@<NSX_MGR>:/tmp/

  3. Enable SSH on NSX Manager and establish an SSH session to the NSX Manager node used in previous step using the root account.

  4. Execute a dry run of the script to identify broken records without making database changes:

    python3 /tmp/fix_missing_parent_path.py

  5. Review the printed list of broken records and computed parent paths.

  6. Apply the database fix:

    python3 /tmp/fix_missing_parent_path.py --apply

    Optional: To perform the fix to specific Transport Zone IDs, use the below command:

    python3 /tmp/fix_missing_parent_path.py --apply-ids <transport_zone_uuid>

    Sample Output:

    root@<NSX_MGR># python3 /tmp/fix_missing_parent_path.py --apply
    Read #### records from nsx$PolicyTransportZone

    Found <no_of_missing_path> record(s) missing parent_path:
      - /infra/sites/default/enforcement-points/default/transport-zones/<transport_zone_uuid>
          displayName : system-owned-vlan-transport-zone-for-rtep
          parent_path to set : /infra/sites/default/enforcement-points/default
          ...

    Applying fixes...
      OK   /infra/sites/default/enforcement-points/default/transport-zones/<transport_zone_uuid> 
      -> parent_path=/infra/sites/default/enforcement-points/default
      ...

    Done. ####/#### record(s) fixed.

  7. Trigger a manual synchronization of the workload domain using Sync Inventory in the VCF Ops UI to verify the resolution.

Attachments

fix_missing_parent_path.py get_app
Powered by Wolken Software