Updating Memory Limits for helm-deployment-server on TMC Self-Managed vCD extension

Products

• VMware Cloud Director
• VMware Tanzu Mission Control - SM

Issue/Introduction

In Tanzu Mission Control Self-Managed (TMC SM) environments deployed via the VMware Cloud Director (vCD) extension, the helm-deployment-server pods may intermittently restart. Investigation often shows a CrashLoopBackOff state caused by out-of-memory (OOMKilled) errors, even though no clear error appears in the pod logs.

Cause

The root cause is typically the default memory resource limit for helm-deployment-server, which is set to 256Mi. This limit may be insufficient for the workloads handled in some environments. 

Resolution

Use the following steps to apply a YTT overlay that increases the memory limit for helm-deployment-server to 512Mi.

1. Create YTT overlay secret for updating resource limits of helm-deployment-server to 512Mi

# kubectl apply -f  helm-deploy-resources-mutator-overlay.yaml

apiVersion: v1
kind: Secret
metadata:
  name: helm-deploy-resources-mutator-overlay
  namespace: tmc-local
stringData:
  helm-deploy-resources-mutator-overlay.yml: |
    #@ load("@ytt:overlay", "overlay")
    #@overlay/match by=overlay.and_op(overlay.subset({"kind": "Deployment"}),  overlay.subset({"metadata": {"name": "helm-deployment-server"}})), expects="1+"
    ---
    spec:
      template:
        spec:
          containers:
            #@overlay/match by=overlay.subset({"name": "helm-deployment-server"}), expects="1+"
            - name: helm-deployment-server
              #@overlay/match missing_ok=True
              resources:
                limits:
                  memory: "512Mi"

2. Create YTT overlay secret for updating packageinstall tmc-local-stack to apply the contents of helm-deploy-resources-mutator-overlay.yml

# kubectl apply -f  tmc-local-stack-overlay.yaml

apiVersion: v1
kind: Secret
metadata:
  name: tmc-local-stack-overlay
  namespace: tmc-local
stringData:
  tmc-local-stack-overlay.yml: |
    #@ load("@ytt:overlay", "overlay")
    #@overlay/match by=overlay.subset({"apiVersion":"packaging.carvel.dev/v1alpha1", "kind":"PackageInstall", "metadata": {"name": "tmc-local-stack"}}),expects="1+"
    ---
    metadata:
      #@overlay/match missing_ok=True
      annotations:
        #@overlay/match missing_ok=True
        ext.packaging.carvel.dev/ytt-paths-from-secret-name.0: helm-deploy-resources-mutator-overlay

3. Patch the packageinstall tanzu-mission-control to apply the contents of tmc-local-stack-overlay.yml

# kubectl patch pkgi tanzu-mission-control --type='merge' -p '{"metadata": {"annotations": {"ext.packaging.carvel.dev/ytt-paths-from-secret-name.0": "tmc-local-stack-overlay"}}}' -n tmc-local

4. Verify the results and status, the memory limits should be updated to 512Mi, and the pods of helm-deployment-server should be restarted and in running status.

# kubectl -n tmc-local get deployment helm-deployment-server -ojsonpath='{.spec.template.spec.containers[0].resources.limits}'

{"memory":"512Mi"}

# kubectl -n tmc-local get pod -l app=helm-deployment-server

NAME                                      READY   STATUS    RESTARTS   AGE
helm-deployment-server-69b95f57d7-bz8bs   1/1     Running   0          30s
helm-deployment-server-69b95f57d7-wmpx7   1/1     Running   0          12s