Kubelet metrics not available in Prometheus tanzu package

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

In TKG v1.5.4, kubelet metrics such as kubelet_volume_stats_capacity_bytes were available from Prometheus but in TKG 1.6 and beyond these metrics are not no longer available.

https://kubernetes.io/docs/reference/instrumentation/metrics/

 

Cause

With the removal of cadvisor daemonset, Prometheus was no longer retrieving the kubelet metrics.

Resolution

The metrics are still available from kubelet and the following scrape config can be added to the prometheus configuration.

 

    - job_name: kubernetes-nodes-kubelet
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - replacement: kubernetes.default.svc:443
        target_label: __address__
      - regex: (.+)
        replacement: /api/v1/nodes/$1/proxy/metrics
        source_labels:
        - __meta_kubernetes_node_name
        target_label: __metrics_path__
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

 

 

See the relevant TKG documentation for details on how to update Prometheus deployment, 2.3 docs.