How to collect metrics from ETCD using Prometheus on TKGm

Products

• VMware

Issue/Introduction

The purpose of this guide is to provide a comprehensive how-to on monitoring ETCD in a way that all configuration will persist in the event of VM deletion/recreation.

Resolution

Preparing Infrastructure

In order for ETCD configuration to survive VM recreation/deletion, we must first assign static IP addresses to the control plane nodes. This can be done by following the guide here.

Preparing ETCD

Each ETCD server exports metrics under the /metrics path on its client port. In order to allow prometheus to access the metrics endpoint, we will need to modify the "--listen-metrics-urls" to include the wildcard IP address: http://0.0.0.0:2381. With this change made, ETCD will listen on all interfaces. To achieve this change we will have to switch context to our management cluster and modify the KubeControlPlane object to include our wildcard IP address. Below are the commands used to carry out this procedure. 

kubectl config get-contexts

kubectl config use-context MGMT-CONTEXT

kubectl get kcp -A

kubectl edit kcp WORKLOAD-CONTROL-PLANE

    etcd:
      local:
        dataDir: /var/lib/etcd
        extraArgs:
          listen-metrics-urls: http://0.0.0.0:2381

Once these changes are made, the ETCD pods will be automatically recreated with the new configuration. 

NOTE: This command will open the config file in vim so make sure to save changes using "wq!".

Preparing Prometheus

You can use the following link here in order to install Prometheus. Before we complete the installation, we must first add a job for monitoring ETCD in the prometheus-data-values.yaml file. In the "targets" field, we must add the static IP addresses for each of our control plane nodes followed by the port 2381.

- job_name: test-etcd
  static_configs:
  - targets: ['10.225.55.216:2381','10.225.55.228:2381','10.225.55.209:2381']

Once this configuration has been made, we can complete the installation.

Conclusion

Once all steps listed above are successfully completed, it will be possible to monitor ETCD metrics on Prometheus. This configuration will persist after VM recreation/deletion.