Telegraf does not get the NodeExporter metrics for the worker nodes

Products

• VMware Cloud PKS

Issue/Introduction

Symptoms:
Telegraf does not get the NodeExporter metrics for the worker nodes in an Enterprise PKS cluster, even though there is a Node-Exporter pod running as a DaemonSet.

Environment

VMware PKS 1.x

Cause

In the Telegraf configuration file for the worker node, there are no Node Exporter endpoints created under the Input configs.

Resolution

This is a known issue affecting Enterprise PKS 1.5. There is currently no resolution.

Workaround:
The following steps can be used as a workaround for this issue (open a port on the worker node and update the telegraf configuration).

Note: This workaround is ephemeral would not persist a cluster recreation or upgrade operation. It is not advisable to create a NodePort service and not recommended to create anything manually in the pks-system namespace. This workaround should be considered to be temporary and only used in the case where there is a requirement for Node exporter metrics.
 

1. Create a service of type NodePort for Node Exporter on the worker node per the following example:

=========
apiVersion: v1
kind: Service
metadata:
name: node-exporter
namespace: pks-system
spec:
type: NodePort
ports:

        port: 9100
        nodePort: 30100
        selector:
        app: node-exporter
        sessionAffinity: None
        =========

2. Create the ClusterMetricSink with the input field set to node-exporter service on the worker node per the following example:

=========
apiVersion: pksapi.io/v1beta1
kind: ClusterMetricSink
metadata:
 name: u-cluster-metric-sink
spec:
 inputs:
 - type: prometheus
   urls:
   - "http://localhost:30100/metrics"
 outputs:
 - type: prometheus_client
   listen: ":9273" `
=========
 

Additional Information

Sink Architecture in Enterprise PKS