TMC upgrade of contour package fails

Products

• VMware Tanzu Mission Control

Issue/Introduction

After successful upgrade of a cluster from 1.23.x to 1.24.x, as next step in the process the contour package installed and managed by TMC have to be upgraded from 

1.18.2 to 1.25.3.

Using TMC UI or ansible script the upgrade have been initiated and failed with below error message:

Useful Error Message:    ytt: Error: Overlaying (in following order: overlays/00-remove-certgen-job.yaml, overlays/01-add-placeholder-secret.yaml, overlays/01-add-psa-label.yaml, overlays/02-update-contour-configmap.yaml, overlays/02-update-contour-deployment.yaml, overlays/02-update-envoy-daemonset.yaml, overlays/02-update-envoy-service.yaml, overlays/02-update-role-contour-psp.yaml, overlays/02-update-role-envoy-psp.yaml, overlays/03-update-namespace.yaml, envoy.yaml):
  Document on line envoy.yaml:4:
    Map item (key 'spec') on line envoy.yaml:5:
      Map item (key 'template') on line envoy.yaml:6:
        Map item (key 'spec') on line envoy.yaml:7:
          Map item (key 'containers') on line envoy.yaml:8:
            Array item on line envoy.yaml:10:
              Map item (key 'livenessProbe') on line envoy.yaml:12:
                Expected number of matched nodes to be 1, but was 0

Environment

TMC as package deployment 

Tanzu on vsphere 

TKGm

Cause

During previous version a custom patch have been applied following below steps:

    1. Create a yaml overlay file for Envoy called envoy.yaml:

#@ load("@ytt:overlay", "overlay")

#@overlay/match by=overlay.subset({"kind": "DaemonSet", "metadata": {"name": "envoy"}})
---
spec:
  template:
    spec:
      containers:
      #@overlay/match by=overlay.subset({"name": "shutdown-manager"})
      -
        #@overlay/remove
        livenessProbe:
#@ load("@ytt:overlay", "overlay")

#@overlay/match by=overlay.subset({"kind": "DaemonSet", "metadata": {"name": "envoy"}})
---
spec:
  template:
    spec:
      containers:
      #@overlay/match by=overlay.subset({"name": "shutdown-manager"})
      -
        #@overlay/match missing_ok=True
        resources:
          requests:
            cpu: "25m"
            memory: "50Mi"

    2.Create a secret for the overlay. For example:

kubectl create secret generic envoy-overlay --from-file=envoy.yaml

    3. Annotate the PackageInstall

kubectl annotate PackageInstall contour ext.packaging.carvel.dev/ytt-paths-from-secret-name.0=envoy-overlay

Once the PKGI is patched is successfully updates the previous version 1.18.2 by removing the liveness probe, however due to 1.25.3 does not have liveness probe the same patch fails with above error

 

Resolution

To mitigate the problem remove the annotation applied the packageinstall 

kubectl annotate PackageInstall contour ext.packaging.carvel.dev/ytt-paths-from-secret-name.0-

Verify if the annotation is removed

kubectl get pkgi  contour -oyaml
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata:
  annotations:
    tanzu.vmware.com/owner: tmc
    tmc.cloud.vmware.com/managed-tanzu-package-install: <ID>/vmware-system-tmc/contour

Additional Information

Problem could be related to other issues, but most likely reason would be custom ytt applied on automated deployment that appears to be unexpected for the new version