Setup resource reservation and limit with Prometheus Add-on

Products

• VMware Telco Cloud Automation

Issue/Introduction

• When deploying Prometheus Add-on via TCA, the cpu and memory resources gets deployed without any reservation or limit

Environment

3.x

Resolution

• Resource limits can be set on TCA UI for Prometheus Addon. Before clicking "DEPLOY CHANGES" button, switch to "Custom Resources (CR)" tab and add the following:
  

  spec:
    name: prometheus
    clusterRef:
      name: ######
      namespace: ######
    config:
      stringData:
        values.yaml: |
          prometheus:
            deployment:
              replicas: 1
              containers:
                resources:
                  limits:
                    cpu: "900m"
                    memory: "6800Mi"
                  requests:
                    cpu: "450m"
                    memory: "6000Mi"
                ...
          alertmanager:
            deployment:
              containers:
                resources:
                  limits:
                    cpu: "3m"
                    memory: "15Mi"
                  requests:
                    cpu: "1m"
                    memory: "13Mi"
              ...
          kube_state_metrics:
            deployment:
              containers:
                resources:
                  limits:
                    cpu: "12m"
                    memory: "90Mi"
                  requests:
                    cpu: "6m"
                    memory: "82Mi"
            ...
          node_exporter:
            daemonset:
              containers:
                resources:
                  limits:
                    cpu: "2.2m"
                    memory: "12Mi"
                  requests:
                    cpu: "1m"
                    memory: "8Mi"
              ...
          pushgateway:
            deployment:
              containers:
                resources:
                  limits:
                    cpu: "1.2m"
                    memory: "8Mi"
                  request:
                    cpu: "1m"
                    memory: "7Mi"
              ...

NOTE: 

The following recommendation is based on an environment without any CNF workload. Numbers might need to be increased based on the environment.

To get the proper numbers based on the environment deploy Prometheus without resource configuration. Periodically, run "kubectl top pod -n tanzu-system-monitoring" command to get cpu and memory usage of each Prometheus pods. Then the max number plus some extra can be used for resource limits.