SDDC Manager upgrade fails at "Apply Configuration drift" with error "DVPG configuration drift parameter preparation failed"

Products

• VMware SDDC Manager

Issue/Introduction

• When upgrading the SDDC Manger, the following issue occurs at  "Apply Configuration drift " task:
  

  Apply Configuration Drift
  Failed Completed at <Date>, <Time>   Started at <Date>, <Time>
  Creates a Distributed Virtual Port Group to enable traffic isolation between management VMs and ESXi hosts
  Error Message: DVPG configuration drift parameter preparation failed
  Remediation message:
  Reference token: <Token>
  Create Service account for SDDC Manager to VxRail Manager communication
  ESXi advanced options for ESXi Hosts Addition Drift 

• In the logs , /var/log/vmware/vcf/domainmanager/domainmanager.log  show reconciliation task fails:
  

  YYYY-MM-DDTHH:MM:SS.sss+0000 DEBUG [vcf_dm,<UUID>] [c.v.v.c.o.s.ConfigreconcilerOrchServiceImpl,dm-exec-14]  Child reconciliation task <TaskID> and drifts [<UUID>, <UUID>, <UUID>] : {"id":"<IDNumber>","name":"Configuration Updates","status":"FAILED","reconciliationSubTasks":[{"driftId":"<ID Number>","driftName":"Creates a Distributed Virtual Port Group to enable traffic isolation between management VMs and ESXi hosts","driftDescription":"Creates a Distributed Virtual Port Group to enable traffic isolation between management VMs and ESXi hosts","status":"FAILED","creationTimestamp":"YYYY-MM-DDTHH:MM:SS.sssZ","completionTimestamp":"YYYY-MM-DDTHH:MM:SS.sssZ","errors":[{"errorCode":"PREPARE_DVPG_CONFIG_DRIFT_PARAM_FAILED","context":{"orchestrator.error.retriable":"false"},"message":"DVPG configuration drift parameter preparation failed","cause":[{"type":"java.lang.IllegalArgumentException","message":"Not found VDS"}],"referenceToken":"87GSO5"}]}]}
  [c.v.v.c.c.a.PrepareDvpgConfigurationParamAction,dm-exec-15]  Not found VDS for cluster <ClusterName>
  YYYY-MM-DDTHH:MM:SS.sss+0000 ERROR [vcf_dm,<IDNumber>] [c.v.v.c.c.a.PrepareDvpgConfigurationParamAction,dm-exec-15]  Error while preparing DVPG configuration drift parameters
  java.lang.IllegalArgumentException: Not found VDS
          at com.vmware.vcf.configreconciler.config.actions.PrepareDvpgConfigurationParamAction.getVDS(PrepareDvpgConfigurationParamAction.java:310)
          at com.vmware.vcf.configreconciler.config.actions.PrepareDvpgConfigurationParamAction.execute(PrepareDvpgConfigurationParamAction.java:245)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
          at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     

• " Get /v1/clusters/{id}/vdses" API returns no output for the cluster, despite DB entries showing two VDS (one of which appears incorrect)
• Network Port groups listed in the database do not align with the actual vCenter configuration
  

  id                 | <IDNumber>
  creation_time      | 1692761664667
  modification_time  | 1692761665388
  mtu                | 0
  name               | vCenter-VDS
  niocs              | 
  port_groups        | [{"name":"vSphere vMotion-<UUID>,"transportType":"VMOTION","type":"EARLY_BINDING","vlanId":###,"sourceId":"dvportgroup-######"}
  status             |
  version            | 7.0.3
  is_used_by_nsxt    | t
  source_id          |  dvs-######
  nsxt_switch_config | ("transportZones":[{"name":"overlay-NSXFQDN","is":"<IDNumber","transportTypr":"OVERLAY"}]}

Cause

The validation error 'Error in validating Config Profile' suggests a mismatch in VDS or port group configurations.

The Network tab for the cluster displays no information, which indicates an incomplete or corrupted association between SDDC Manager and the vCenter inventory

Resolution

1. Take a snapshot of the SDDC manager appliance.
   
   
2. SSH into the SDDC manager with vcf and then elevate to root. 
   
   
3. Connect to the SDDC DB.
   
   psql -h localhost -U postgres -d platform 
   
   
4. Identify the VDS id with the missing port group with the below query.
   
   select id,name,port_groups from vds;


5. To update the VDS entries
   
   Use SQL commands to Update name, transportType, vlanID (if applicable), uplink configuration, source ID for each port group that is missing.
   
   Note: Contact Broadcom Support for assistance.
   
   
6. Restart services on SDDC Manager and vCenter to ensure changes take effect.
   
   
   • To restart SDDC manager services:

/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

   • To restart vCenter server services:
     
      service-control --stop --all && service-control --start --all
     
     
7. Try running the upgrade again.