SDDC Manager Config drift upgrade failed to ADD_EDGE_TEAMING_STANDBY_UPLINK_UPDATE_FAILED
search cancel

SDDC Manager Config drift upgrade failed to ADD_EDGE_TEAMING_STANDBY_UPLINK_UPDATE_FAILED

book

Article ID: 316057

calendar_today

Updated On:

Products

VMware Cloud Foundation VMware SDDC Manager

Issue/Introduction

  • This article defines the workaround steps to proceed with SDDC Manager config drift upgrade when it fails because of an unknown Portgroup ID for a Distributed Virtual Port group (DVPG)
  • SDDC Manager Config drift upgrade failed to ADD_EDGE_TEAMING_STANDBY_UPLINK_UPDATE_FAILED
  • Error in /var/log/vmware/vcf/lcm/thirdparty/upgrade/<latest upgrade id>/logs/sddc_manager_migration/sddc_manager_xxxxx.log 
    Failed to add standby uplink to edge cluster xxxxxxxx portgroup (unknown)

ERROR [vcf_migration,0000000000000000,0000] [c.v.e.s.c.c.v.vsphere.VcManagerBase,pool-5-thread-14]  Entity DistributedVirtualPortgroup:/infra/segments/xxxxxxxx not found:
com.vmware.vim.binding.vmodl.fault.ManagedObjectNotFound: The object 'vim.ManagedEntity:/infra/segments/xxxxxxx' has already been deleted or has not been completely created

ERROR [vcf_migration,0000000000000000,0000] [c.v.v.m.a.AddPortgroupTeamingStandbyUplink,pool-5-thread-14]  Unknown portgroup MoId /infra/segments/xxxxxxx

ERROR [vcf_migration,0000000000000000,0000] [c.v.e.s.o.model.error.ErrorFactory,pool-5-thread-14]  [SVU14H] ADD_EDGE_TEAMING_STANDBY_UPLINK_UPDATE_FAILED Failed to add standby uplink to edge cluster xxxxxxx portgroup (unknown)
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to add standby uplink to edge cluster xxxxxxxx portgroup (unknown)
   
Caused by: java.lang.Exception: Unknown portgroup MoId /infra/segments/xxxxxxxx

Environment

VMware Cloud Foundation 4.5.1

Cause

  • Duplicate or stale Distributed Virtual Port group (DVPG).
  • Changes performed to the DVPG outside of SDDC Manager

Resolution

This issue is resolved in VCF 5.0 onwards as NSXT Edge drift is removed from SDDC config drift and will no longer run.

 

Workaround:

Remove the NSXT Edge Cluster from SDDC Inventory and restore after a successful config drift upgrade

 

Below are the steps to remove and restore the NSXT Edge cluster from SDDC Inventory

 

  1. Take snapshot of SDDC Manager VM (DO NOT proceed without this step)
  2. SSH to SDDC Manager with vcf user and su to root
  3. Query the nsxt_edge_cluster 
    psql -h localhost -U postgres -d platform -c "\x" -c "select * from nsxt_edge_cluster"

    Sample output

    id                          | 2b43####-####-####-####-########80ce
creation_time        | 1701282212235
modification_time  | 1701287725435
status                     | ACTIVE
name                      | myCluster
nsxt_edge_nodes   | [{"vmManagementIpAddress":"10.0.0.51","vmHostname":"nsxt-edge-node-2.example.com","sourceId":"91d0####-####-####-####-########b10e","id":"1a13####-####-####-####-########7483"},{"vmManagementIpAddress":"10.0.0.50","vmHostname":"nsxt-edge-node-1.example.com","sourceId":"1887####-####-####-####-########3ee5","id":"d16b####-####-####-####-########c404"},{"vmManagementIpAddress":"10.0.0.54","vmHostname":"nsxt-edge-node-5.example.com","sourceId":"d8ed####-####-####-####-########0e8f","id":"576b####-####-####-####-########706f"}]
source_id                 | 458e####-####-####-####-########b120
     
  4. Take a dump from nsxt_edge_cluster and nsxt_edge_cluster_and_nsxt_cluster  tables into SQL file. 
    pg_dump --table=nsxt_edge_cluster_and_nsxt_cluster --data-only --column-inserts platform -U postgres -h localhost > data1.sql
    pg_dump --table=nsxt_edge_cluster --data-only --column-inserts platform -U postgres -h localhost > data2.sql
  5. Note down the INSERT string from data1.sql and data2.sql which will be required to restore the edge clusters after successful config drift upgrade
    • vi data1.sql 
      insert into public.nsxt_edge_cluster (id, creation_time, modification_time, status, name, nsxt_edge_nodes, source_id, is_tier0managed_by_system, skip_tep_routability_check) values ('2b43####-####-####-####-########80ce',1701282212235,1701287725435,'ACTIVE','myCluster', '[{"vmManagementIpAddress":"10.0.0.51","vmHostname":"nsxt-edge-node-2.example.com","sourceId":"91d0####-####-####-####-########b10e","id":"1a13####-####-####-####-########7483"},{"vmManagementIpAddress":"10.0.0.54","vmHostname":"nsxt-edge-node-5.example.com","sourceId":"d8ed####-####-####-####-########0e8f","id":"576b####-####-####-####-########706f"}]', '458e####-####-####-####-########b120',t,f);
       
    • vi data2.sql

      insert into public.nsxt_edge_cluster_and_nsxt_cluster (nsxt_edge_cluster_id, nsxt_cluster_id) values (2b43####-####-####-####-########80ce, 3585####-####-####-####-########99a2);
  6. Delete the Edge cluster from SDDC Inventory 
    psql -h localhost -U postgres -d platform -c "delete from nsxt_edge_cluster where id='<nsxt_edge_cluster id gathered in step # 3>"
    psql -h localhost -U postgres -d platform -c "delete from nsxt_edge_cluster_and_nsxt_cluster where nsxt_edge_cluster_id=<nsxt_edge_cluster id gathered in step # 3>"
  7. Retry the SDDC Config drift upgrade
  8. Once successfully finished upgrade, restore db tables by copying insert statements from data2.sql and data1.sql 
    psql -h localhost -U postgres -d platform -c "<INSERT string gathered from data1.sql in Step#5>"
    psql -h localhost -U postgres -d platform -c "<INSERT string gathered from data2.sql in Step#5>"


    Note: Remove the schema "public" from the INSERT string before executing
Powered by Wolken Software