Error: "ReconcileFailed. Message: kapp: Error: waiting on reconcile deployment/contour" when enabling the Contour Supervisor service

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

•  "Link Down" for the pods is seen on the UI. 

•  Pod description :

Events:
  Type     Reason                     Age                    From                    Message
  ----     ------                     ----                   ----                    -------
  Normal   Scheduled                  6m13s                  default-scheduler       Successfully assigned svc-contour-domain-cX/contour-xxxxxxxxx-xxxxx to <esxi-hostname>
  Normal   Synced                     6m11s                  pod-network-controller  Pod network successfully bound to pod
  Warning  FailedMount                6m11s                  kubelet                 failed to mount volume contourcert: error retrieving Secret contourcert: secret "contourcert" not found
  Normal   PodVMNetworkConfigSuccess  6m11s                  scheduler-extender      Added network interface device backed by dvportgroup-xxxxx Network to Pod
  Normal   NetworkInterfaceRemoved    6m10s (x2 over 6m10s)  pod-network-controller  NetworkInterface 'svc-contour-domain-cx/vmware-system-pod-<UUID>' removed


•  Configuration error on UI :

Reason: ReconcileFailed. Message: kapp: Error: waiting on reconcile deployment/contour (apps/v1) namespace: svc-contour-domain-cx: Finished unsuccessfully (Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "contour-xxxxxxxxxx" has timed out progressing.)).

• From vCenter Server you can see similar log entries under location /var/log/vmware/vpxd/vpxd.log

YYYY-MM-DDThh:mm:ss.Z info vpxd[06078] [Originator@6876 sub=vpxLro opID=29a1b374] [VpxLRO] -- BEGIN session[<UUID>]<UUID>-- dvportgroup-xx -- vim.dvs.DistributedVirtualPortgroup.allocate -- <UUID>(<UUID>)
YYYY-MM-DDThh:mm:ss.Z  warning vpxd[06078] [Originator@6876 sub=MoDVPortGroup opID=29a1b374] podVM: allocate port not supported on nsx dvpg.
YYYY-MM-DDThh:mm:ss.Z  warning vpxd[06078] [Originator@6876 sub=Vmomi opID=29a1b374] VMOMI activation LRO failed; <<<UUID>, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 46782'>>, dvportgroup-xx, vim.dvs.DistributedVirtualPortgroup.allocate, <vim.version.v8_0_2_0, official, 8.0.2.0>, (null)>, N5Vmomi5Fault15InvalidArgument9ExceptionE(Fault cause: vmodl.fault.InvalidArgument
--> )
--> [context]………..[/context]
YYYY-MM-DDThh:mm:ss.Z  info vpxd[06078] [Originator@6876 sub=vpxLro opID=29a1b374] [VpxLRO] -- FINISH session[<UUID>]<UUID>
YYYY-MM-DDThh:mm:ss.Z error vpxd[06078] [Originator@6876 sub=Default opID=29a1b374] [VpxLRO] -- ERROR session[<UUID>]<UUID>-- <UUID>(<UUID>) -- dvportgroup-xx -- vim.dvs.DistributedVirtualPortgroup.allocate: :vmodl.fault.InvalidArgument
--> Result:
--> (vmodl.fault.InvalidArgument) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.vim.vpxd.dvs.invalidActionForNsxPortgroup.label",
-->          arg = <unset>,
-->          message = <unset>
-->       }
-->    ],
-->    invalidProperty = "DistributedVirtualPortgroup"
-->    msg = ""
--> }
--> Args:
-->
--> "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
--> Arg host:
--> 'vim.HostSystem:host-XXXXX'
YYYY-MM-DDThh:mm:ss.Z  info vpxd[06105] [Originator@6876 sub=vpxLro opID=wcp-licenseRefreshMonitor-10] [VpxLRO] -- BEGIN lro-15460590 -- LicenseAssignmentManager -- vim.LicenseAssignmentManager.isFeatureAvailable -- <UUID>(<UUID>)
YYYY-MM-DDThh:mm:ss.Z  info vpxd[06105] [Originator@6876 sub=vpxLro opID=wcp-licenseRefreshMonitor-10] [VpxLRO] -- FINISH lro-15460590

Environment

vSphere with Tanzu 8

Cause

Workload Management (WCP) was enabled using a VDS networking stack (HAProxy/ALB) and was configured to use portgroups managed by NSX-T during the WCP configuration wizard.

Resolution

Enable WCP using a VDS network stack (HAproxy/ALB) and select a non-managed NSX-T portgroup when enabling WCP or enable WCP using NSX-T as the network provider.