Allowing metrics HTTP endpoints on Tanzu Kubernetes Grid (TKG) control plane components such as kube-scheduler and kube-controller-manager
search cancel

Allowing metrics HTTP endpoints on Tanzu Kubernetes Grid (TKG) control plane components such as kube-scheduler and kube-controller-manager

book

Article ID: 345691

calendar_today

Updated On:

Products

VMware

Issue/Introduction

Some Kubernetes components, such as the kube-scheduler and kube-controller-manager, provide the /metrics endpoint only on localhost/127.0.0.1 by default. 

The purpose of this article is to provide steps to expose the /metrics endpoint to be reachable by other nodes.


Resolution

Summary

In the below procedure, a TKG cluster is first created. Next, the Prometheus extension is deployed using the Deploy Prometheus on Tanzu Kubernetes Clusters documentation. Next, an overlay is created to reassign the parameter "--port=0" to a non-zero value on the kube-controller-manager and/or kube-scheduler. Finally, a new TKG cluster is created using the existing TKG cluster with the overlay.


Procedure

1. Complete all of the prerequisites mentioned in the Deploy Prometheus on Tanzu Kubernetes Clusters documentation.

2. Confirm the TKG cluster and Prometheus extension is running normally.

3. From the bootstrap host of the running TKG cluster, make a copy of the TKG cluster clusterconfig:

\cp ~/.tanzu/tkg/clusterconfigs/CLUSTER-NAME.yaml  /tmp/NEW-CLUSTER-NAME.yaml

4. Edit /tmp/NEW-CLUSTER-NAME.yaml and update the variables for your new cluster.  In many cases, you only need to change the CLUSTER_NAME and VSPHERE_CONTROL_PLANE_ENDPOINT values. 

Example modifications:

\CLUSTER_NAME: "NEW-CLUSTER-NAME"
VSPHERE_CONTROL_PLANE_ENDPOINT: "<New Static IP>"


5. Create the overlay yaml file:

# Where PLATFORM is be one of aws, azure or vsphere

cd ~/.tanzu/tkg/providers/infrastructure-PLATFORM/ytt/

vim PLATFORM-overlay-assign-metrics-ep-port.yaml


6. To create an overlay which exposes both kube-controller-manager and kube-scheduler /metrics endpoints, add the following contents to PLATFORM-overlay-assign-metrics-ep-port.yaml
:

#@ load("@ytt:overlay", "overlay") #@overlay/match by=overlay.subset({"kind":"KubeadmControlPlane"}),expects="0+"
---
spec:
   kubeadmConfigSpec:
      clusterConfiguration:
         controllerManager:
            extraArgs: #@overlay/match missing_ok=True
            port: "10252" scheduler:
               extraArgs:
                  #@overlay/match missing_ok=True port: "10251"


7. Save the file.

8. Create the new cluster using the "tanzu" CLI (as of TKG 1.3):

 tanzu cluster create NEW-CLUSTER-NAME --file /tmp/NEW-CLUSTER-NAME.yaml --verbose 9