Preparing InfrastructureIn order for ETCD configuration to survive VM recreation/deletion, we must first assign static IP addresses to the control plane nodes. This can be done by following the guide
here.
Preparing ETCDEach ETCD server exports metrics under the
/metrics
path on its client port. In order to allow prometheus to access the metrics endpoint, we will need to modify the "--listen-metrics-urls" to include the wildcard IP address: http://0.0.0.0:2381. With this change made, ETCD will listen on all interfaces. To achieve this change we will have to switch context to our management cluster and modify the KubeControlPlane object to include our wildcard IP address. Below are the commands used to carry out this procedure.
kubectl config get-contexts
kubectl config use-context MGMT-CONTEXT
kubectl get kcp -A
kubectl edit kcp WORKLOAD-CONTROL-PLANE
etcd:
local:
dataDir: /var/lib/etcd
extraArgs:
listen-metrics-urls: http:
//0.0.0.0:2381
Once these changes are made, the ETCD pods will be automatically recreated with the new configuration.
NOTE: This command will open the config file in vim so make sure to save changes using "wq!".
Preparing PrometheusYou can use the following link
here in order to install Prometheus. Before we complete the installation, we must first add a job for monitoring ETCD in the prometheus-data-values.yaml file. In the "targets" field, we must add the static IP addresses for each of our control plane nodes followed by the port 2381.
- job_name: test-etcd
static_configs:
- targets: ['10.225.55.216:2381','10.225.55.228:2381','10.225.55.209:2381']
Once this configuration has been made, we can complete the installation.
ConclusionOnce all steps listed above are successfully completed, it will be possible to monitor ETCD metrics on Prometheus. This configuration will persist after VM recreation/deletion.