Wavefront observability-system namespace gets deleted when updating/upgrading TKGI clusters

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

If you have deployed the Wavefront Observability for Kubernetes Operator manually on a TKGI cluster, it may get deleted when updating/upgrading the cluster.

This issue is known to happen with the following configuration:

1. You have deployed the Observability for Kubernetes Operator via some process external to TKGI
2. Wavefront Integration is disabled in the TKGI tile settings

Environment

TKGI v1.20.0 or later

Cause

When you update or upgrade TKGI clusters, it runs the wavefront-proxy-errand. This errand deletes the observability-system namespace if Wavefront Integration is disabled in the TKGI tile settings

Resolution

One solution is to enable the TKGI Wavefront integration in the tile settings instead of deploying the Wavefront operator yourself. Please note the TKGI Wavefront Integration setting is deprecated and will be removed in a future release

Another solution is to use the following runtime config to patch the wavefront-proxy-errand so it doesn't delete the observability-system namespace:

• Upload os-conf release to bosh (we have tested this patch with os-conf v23.0.0)

  • bosh upload-release "https://bosh.io/d/github.com/cloudfoundry/os-conf-release?v=23.0.0"

• Save the following runtime config to a file (wavefront-errand-config.yml for example)

  • addons:
    - name: wavefront-errand-config
      jobs:
      - name: pre-start-script
        release: os-conf
        properties:
          script: |-
            #!/bin/bash
            if [ -f /var/vcap/jobs/wavefront-proxy-errand/bin/run ]; then
              sed -i '/delete_spec "wavefront.yml"/d' /var/vcap/jobs/wavefront-proxy-errand/bin/run
              sed -i '/delete_spec "wavefront-proxy.yml"/d' /var/vcap/jobs/wavefront-proxy-errand/bin/run
              sed -i '/delete_spec "wavefront-operator.yml"/d' /var/vcap/jobs/wavefront-proxy-errand/bin/run
              sed -i '/${kubectl} delete secret wavefront-secret --namespace=${new_namespace} --ignore-not-found/s/^/# /' /var/vcap/jobs/wavefront-proxy-errand/bin/run
            fi
      include:
        instance_groups: [apply-addons]
    releases:
    - name: "os-conf"
      version: "((OS_CONF_RELEASE))"

• Upload the runtime config to bosh

  • bosh update-runtime-config --name wavefront-errand-config ./wavefront-errand-config.yml

• Try your cluster update/upgrade again and verify wavefront deployment is not deleted