SDDC Manager upgrade failed at apply configuration drift | DVPG configuration drift parameter preparation failed

Products

VMware SDDC Manager

Issue/Introduction

When upgrading the SDDC Manger, the following occurs at the Apply Configuration Update task:

Apply Configuration Update
Failed Completed at <Date>, <Time> Started at <Date>, <Time>
Applying 3 Configuration Updates
Creates a Distributed Virtual Port Group to enable traffic isolation between management VMs and ESXi hosts
Error Message: DVPG configuration drift parameter preparation failed
Remediation message:
Reference token: <Token>
Create Service account for SDDC Manager to VxRail Manager communication
ESXi advanced options for ESXi Hosts Addition Drift

Domain Manager logs show Reconciliation task fails:

YYYY-MM-DDTHH:MM:SS.sss+0000 DEBUG [vcf_dm,<UUID>] [c.v.v.c.o.s.ConfigreconcilerOrchServiceImpl,dm-exec-14] Child reconciliation task <TaskID> and drifts [<UUID>, <UUID>, <UUID>] : {"id":"<IDNumber>","name":"Configuration Updates","status":"FAILED","reconciliationSubTasks":[{"driftId":"<ID Number>","driftName":"Creates a Distributed Virtual Port Group to enable traffic isolation between management VMs and ESXi hosts","driftDescription":"Creates a Distributed Virtual Port Group to enable traffic isolation between management VMs and ESXi hosts","status":"FAILED","creationTimestamp":"YYYY-MM-DDTHH:MM:SS.sssZ","completionTimestamp":"YYYY-MM-DDTHH:MM:SS.sssZ","errors":[{"errorCode":"PREPARE_DVPG_CONFIG_DRIFT_PARAM_FAILED","context":{"orchestrator.error.retriable":"false"},"message":"DVPG configuration drift parameter preparation failed","cause":[{"type":"java.lang.IllegalArgumentException","message":"Not found VDS"}],"referenceToken":"87GSO5"}]}]}

[c.v.v.c.c.a.PrepareDvpgConfigurationParamAction,dm-exec-15] Not found VDS for cluster <ClusterName>
YYYY-MM-DDTHH:MM:SS.sss+0000 ERROR [vcf_dm,<IDNumber>] [c.v.v.c.c.a.PrepareDvpgConfigurationParamAction,dm-exec-15] Error while preparing DVPG configuration drift parameters
java.lang.IllegalArgumentException: Not found VDS
at com.vmware.vcf.configreconciler.config.actions.PrepareDvpgConfigurationParamAction.getVDS(PrepareDvpgConfigurationParamAction.java:310)
at com.vmware.vcf.configreconciler.config.actions.PrepareDvpgConfigurationParamAction.execute(PrepareDvpgConfigurationParamAction.java:245)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751)
at org.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:163)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703)
at com.vmware.vcf.configreconciler.config.actions.PrepareDvpgConfigurationParamAction$$SpringCGLIB$$0.execute(<generated>)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:159)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:144)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:400)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:520)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:124)
at jdk.internal.reflect.GeneratedMethodAccessor818.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85)
at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71)
at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:59)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

Environment

SDDC Manager

Cause

1. The error 'Error in validating Config Profile'` in the switch configuration step suggests a mismatch in the VDS or port group configurations.
2. The network tab for the cluster shows no information, indicating an incomplete or corrupted association between the SDDC Manager and vCenter inventory.
3. The '/v1/clusters/{id}/vdses' API returns no output for the cluster, despite DB entries showing two VDS (one of which appears incorrect).
Port groups listed in the database do not align with the actual vCenter configuration.

id | <IDNumber>
creation_time | 1692761664667
modification_time | 1692761665388
mtu | 0
name | XX-Vcenter-VDSXX
niocs |
port_groups | [{"name":"vSphere vMotion-<UUID>,"transportType":"VMOTION","type":"EARLY_BINDING","vlanId":###,"sourceId":"dvportgroup-######"}
status |
version | 7.0.3
is_used_by_nsxt | t
source_id | dvs-######
nsxt_switch_config | ("transportZones":[{"name":"overlay-NSXFQDN","is":"<IDNumber","transportTypr":"OVERLAY"}]}
-[ RECORD 2 ]------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

In this case, only "vMotion" is populated in the port group. In vCenter, it is assigned Management, vMotion, vSAN, and VM_Management, etc.

[c.v.e.s.c.s.ClusterNetworkConfigFetcher,dm-exec-1] Portgroup of VDS <vCenter-vds01> from VC [Manage
ment Network-<IDNumber>, Virtual SAN-<UUID>, Virtual SAN-<UUID>, vSphere vMotion-<UUID>, Virtual SAN-<UUID>, NSX-T-Overlay, VxRail Management-<UUID>, vCenter Server Network-<UUID>, VMware HCIA Dist-DVUplinks-#####]

Resolution

Take a snapshot of the SDDC manager appliance.
SSH into the SDDC manager with vcf and then elevate to root.
Connect to the SDDC DB.

psql -h localhost -U postgres -d platform
Identify the VDS id with the missing port group with the below query.

select id,name,port_groups from vds;
Update the VDS Entries

Use SQL commands to Update name, transportType, vlanID (if applicable), uplink configuration, sourceid for each port group that is missing.
Please ensure that an SDDC TSE implements or, at least, supervises these changes.
Restart services on SDDC Manager and vCenter to ensure changes take effect.

To restart SDDC manager services:

/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh
Try running the upgrade again.