Missing kube-scheduler metrics kube_pod_resource_request and kube_pod_resource_limit

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

"Include Kubernetes Scheduler metrics" enabled in TKGi Tile–Host Monitoring and kube-scheduler metrics are collected. However resource metrics kube_pod_resource_request and kube_pod_resource_limit are not present.

Environment

TKGi 1.22

Cause

The input plugin in /var/vcap/jobs/metric-sink/config/telegraf.conf is configured to gather metrics from https://localhost:10259/metrics but the resource metrics are available from https://localhost:10259/metrics/resources 

Resolution

A fix will be available in a future patch release, please contact Tanzu Support for details.

The following steps can be used as a workaround but please note that the changes are not persistent.

Update the system:monitoring ClusterRole to allow access the /metrics/resources

kubectl patch clusterrole system:monitoring --type='json' -p='[
  {
    "op": "add",
    "path": "/rules/0/nonResourceURLs/5",
    "value": "/metrics/resources"
  }
]'

Add the following to Inputs Plugins section of /var/vcap/jobs/metrics-sink/config/telegraf.conf

[[inputs.prometheus]]
  urls = ["https://localhost:10259/metrics/resources"]
  metric_version = 2
  alias = "scheduler"
  insecure_skip_verify = true
  tls_cert = "/var/vcap/jobs/metric-sink/config/monitoring_metric_cert.pem"
  tls_key = "/var/vcap/jobs/metric-sink/config/monitoring_metric_cert.key"

Restart telegraf

monit restart telegraf
monit summary