How to scrape Control plane component and ETCD metrics to Prometheus in Enterprise PKS

Products

• VMware Cloud PKS

Issue/Introduction

Symptoms:
When Prometheus is Integrated with Enterprise PKS, no metrics for Control plane components and ETCD are received.

Environment

VMware PKS 1.x

Cause

In Enterprise PKS, Control plane components and ETCD process are running as a BOSH Monit process so node exporter will not collect the metrics for these process by default.

Resolution

Below should be the prerequisites:

• K8s Cluster
• Default StorageClass (optional, only if using persistent storage)
• Helm client/server version =>2.13.1
• Assumes BOSH knowledge

Steps to Deploy Prometheus Operator on Enterprise PKS:

Login to a VM where you can run bosh commands:
 

1. Get the IP of Master node:

bosh vms

2. Get ETCD certs from Master(s)

bosh scp -d service-instance_<####> master/0:/var/vcap/jobs/etcd/config/etcd-ca.crt .
bosh scp -d service-instance_<####> master/0:/var/vcap/jobs/etcd/config/etcdctl.* .

Note: Login to a VM where you can authenticate to your clusters so that you can run kubectl commands to interact with the K8s clusters.

3. Create a monitoring namespace for the operator to live in

kubectl create ns monitoring

4. Create ETCD certs secret in K8s using the certs copied from the Master

kubectl create secret -n monitoring generic etcd-client \
--from-file=etcd-ca.crt \
--from-file=etcdctl.crt \
--from-file=etcdctl.key

5. Create a file named override.yaml with the following content

prometheus:
  prometheusSpec:
    secrets:
      - etcd-client
kubelet:
  serviceMonitor:
    https: true
kubeControllerManager:
  endpoints:
    - 172.15.0.2     ## Change to the Master IP(s) recorded from "bosh vms" command
kubeScheduler:
  endpoints:
    - 172.15.0.2     ## Change to the Master IP(s) recorded from "bosh vms" command
kubeEtcd:
  endpoints:
    - 172.15.0.2     ## Change to the Master IP(s) recorded from "bosh vms" command
  serviceMonitor:
    insecureSkipVerify: true
    scheme: https
    caFile: "/etc/prometheus/secrets/etcd-client/etcd-ca.crt"
    certFile: "/etc/prometheus/secrets/etcd-client/etcdctl.crt"
    keyFile: "/etc/prometheus/secrets/etcd-client/etcdctl.key"

 

6. Install the prometheus-operator using the following command:

helm install -f override.yaml --name prom-operator stable/prometheus-operator

Additional Information

To Delete the helm chart:

helm delete prom-operator --purge

Along with chart deletion you will need to delete the corresponding CRDs that it has created:
 

kubectl delete crd prometheuses.monitoring.coreos.com
kubectl delete crd prometheusrules.monitoring.coreos.com
kubectl delete crd servicemonitors.monitoring.coreos.com
kubectl delete crd alertmanagers.monitoring.coreos.com
kubectl delete crd podmonitors.monitoring.coreos.com