Specifications applied to tkgserviceconfiguration reverts back in Tanzu Kubernetes Grid Service 3.0, 3.1

Products

• VMware vSphere with Tanzu

Issue/Introduction

• TkgServiceConfiguration Specification Parameter will revert to default if the version of Tanzu Kubernetes Grid Service is 3.0 or 3.1. 
  
  
  
• This will remove the parameters like trust, additionalTrustedCAs, httpProxy, etc. configured as per: https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere-supervisor/7-0/vsphere-with-tanzu-configuration-and-management-7-0.html

Environment

VMware vSphere with Tanzu: >= v80u3
TKG Service: v3.0.0 and v3.1.0

Cause

As part of moving to TKG as a Supervisor Service, the default TKGServiceconfiguration was bundled within a Carvel package.
Since it's an object managed by Kapp, any edits will be overridden during the next package/app reconciliation, unless there are rebase rules.

Resolution

The recommended fix is to update to TKG Service 3.2 version that contains the fix. 

Please the below steps to workaround the issue without an upgrade by adding rebase rules.

Step 1: Create a secret in the TKG Service namespace

This secret is used to update the tkg-service-configuration-kapp-config ConfigMap containing rebase rules for TkgServiceConfiguration kind objects.
The object exists in the TKG Service's Namespace.

tkgserviceconfiguration-rebase-rule-fix.yaml

apiVersion: v1

kind: Secret

metadata:

  name: tkgserviceconfiguration-rebase-rule-fix

  # namespace: svc-tkg-domain-cXX Update this with the TKG Service namespace

stringData:

  rebase-rule-fix.yaml: |

    #@ load("@ytt:overlay", "overlay")

    #@overlay/match by=overlay.subset({"kind":"ConfigMap", "metadata": {"name": "tkg-service-configuration-kapp-config"}}), expects="0+"

    ---

    kind: ConfigMap

    data:

      #@overlay/match missing_ok=True

      #@yaml/text-templated-strings

      config.yml: |

        apiVersion: kapp.k14s.io/v1alpha1

        kind: Config

        rebaseRules:

          - path: [spec]

            type: copy

            sources: [existing, new]

            resourceMatchers:

              - apiVersionKindMatcher:

                  apiVersion: run.tanzu.vmware.com/v1alpha3

                  kind: TkgServiceConfiguration

 Collapse source

CPVM

# Create file from yaml above

$ vi tkgserviceconfiguration-rebase-rule-fix.yaml

 

# Get TKG SVS namespace

$ TKG_NS=$(kubectl get ns | grep svc-tkg | gawk '{print $1}')

 

# Confirm this is the namespace with the configmap we want to overlay

$ kubectl get -n ${TKG_NS} configmap tkg-service-configuration-kapp-config

 

# Create the secret in the TKG service namespace 

$ kubectl apply -f tkgserviceconfiguration-rebase-rule-fix.yaml -n ${TKG_NS}

Step 2: Create a secret in vmware-system-supervisor-services namespace

This secret is used to update the packageInstall for tkgs-config to add a new ytt secret path.

tkgs-config-overlay-fix.yaml

apiVersion: v1

kind: Secret

metadata:

  #  ext.packaging.carvel.dev/ytt-paths-from-secret-name.99: tkgs-config-overlay-fix

  name: tkgs-config-overlay-fix

  namespace: vmware-system-supervisor-services

stringData:

  overlay-fix.yaml: |

    #@ load("@ytt:overlay", "overlay")

    #! This is used in placeholder-util-secret.yaml

    #@overlay/match by=overlay.subset({"kind":"PackageInstall", "metadata": {"name": "tkgs-config"}}), expects="0+"

    ---

    kind: PackageInstall

    metadata:

      #@overlay/match missing_ok=True

      annotations:

        #@overlay/match missing_ok=True

        ext.packaging.carvel.dev/ytt-paths-from-secret-name.98: tkgserviceconfiguration-rebase-rule-fix

 Collapse source

CPVM

# Create file from yaml above

$ vi tkgs-config-overlay-fix.yaml

 

# Create the secret in the vmware-system-supervisor-services namespace

$ kubectl apply -f tkgs-config-overlay-fix.yaml

Step 3: Annotate the svc-tkg PackageInstall with the ytt secret 

This step annotates the outer TKG Service package.

CPVM

$ kubectl annotate pkgi -n vmware-system-supervisor-services svc-tkg.vsphere.vmware.com ext.packaging.carvel.dev/ytt-paths-from-secret-name.99=tkgs-config-overlay-fix

Step 4: Confirm the tkg-service-configuration-kapp-config ConfigMap is updated

Verify the rules get applied

CPVM

# Confirm packages are reconciled

$ kubectl  get app,pkgi -n vmware-system-supervisor-services svc-tkg.vsphere.vmware.com

NAME                            DESCRIPTION           SINCE-DEPLOY   AGE

svc-tkg.vsphere.vmware.com      Reconcile succeeded   10m            3d7h

 

NAME                                                                PACKAGE NAME                PACKAGE VERSION           DESCRIPTION           AGE

packageinstall.packaging.carvel.dev/svc-tkg.vsphere.vmware.com      tkg.vsphere.vmware.com      3.0.0-embedded            Reconcile succeeded   3d7h

 

$ kubectl  get app,pkgi -n ${TKG_NS} tkgs-config

NAME                            DESCRIPTION           SINCE-DEPLOY   AGE

tkgs-config                     Reconcile succeeded   10m            3d7h

 

 

NAME                                      PACKAGE NAME                PACKAGE VERSION           DESCRIPTION           AGE

tkgs-config                               tkgs-config-vsphere.tanzu.vmware.com                 3.0.0             Reconcile succeeded   3d7h

 

 

# Confirm the ytt overlay applied correctly

$ kubectl get configmap -n ${TKG_NS} tkg-service-configuration-kapp-config -oyaml

apiVersion: v1

data:

  config.yml: |

    apiVersion: kapp.k14s.io/v1alpha1

    kind: Config

    rebaseRules:

      - path: [spec] # <--------------------------- this is the field to check

        type: copy

        sources: [existing, new]

        resourceMatchers:

          - apiVersionKindMatcher:

              apiVersion: run.tanzu.vmware.com/v1alpha3

              kind: TkgServiceConfiguration

kind: ConfigMap

metadata:

  name: tkg-service-configuration-kapp-config

  namespace: svc-tkg-domain-cXX

Verify workaround 

To verify the workaround make edits to TkgServiceConfiguration and confirm they are retained for longer than 10 minutes. 

https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere-supervisor/8-0/using-tkg-service-with-vsphere-supervisor.html

Manually removing workaround 

The workaround should be removed after moving to a version that fixes this issue (v3.2.0).

Additional Information

Service Upgrade

As part of Service upgrades, app platform only patches the spec of TKG Service's PackageInstall and this workaround will be carried forward until manually removed.

CPVM

# Confirm TKG Service version contains the fix:

$ kubectl  get app,pkgi -n vmware-system-supervisor-services svc-tkg.vsphere.vmware.com

 

# Remove annotation

$ kubectl annotate pkgi -n vmware-system-supervisor-services svc-tkg.vsphere.vmware.com ext.packaging.carvel.dev/ytt-paths-from-secret-name.99-

 

# Delete secrets

$ kubectl delete secret -n vmware-system-supervisor-services tkgs-config-overlay-fix

$ kubectl delete secret -n ${TKG_NS} tkgserviceconfiguration-rebase-rule-fix