Modify the cluster Autoscaler Configuration Options in an existing Tanzu Kubernetes Grid cluster

Products

• Tanzu Kubernetes Grid

Issue/Introduction

This article provides steps on how to modify the autoscaler config for an existing VMware Tanzu Kubernetes Grid cluster

Symptoms:

• According to the VMware Tanzu Kubernetes documentation  Grid we cannot modify the Autoscaler configuration parameters. after you deploy the cluster. 

Environment

VMware Tanzu Kubernetes Grid 1.x

Resolution

tanzu cli currently does not support modifying autoscaler settings. Please refer to the workaround section for these instructions.

Workaround:

Depending on the parameter that you are trying to modify there are two Kubernetes object where this configuration can be changed. Please make sure that the kubectl context is changed to management cluster before running the commands below

 

• To modify the AUTOSCALER_MIN_SIZE_X and the AUTOSCALER_MAX_SIZE_X you will need to update the cluster "machinedeployments" that handles the worker nodes with  the new values 

1. Run the following command  to get the cluster machinedeployments.

kubectl get machinedeployments
NAME               PHASE     REPLICAS   READY   UPDATED   UNAVAILABLE
cluster-one-md-0   Running   1          1       1
tkg-wld-md-0       Running   1          1       1
 

2. Run the following command to Edit the cluster machinedeployment that we want to modify it autoscaler config.

kubectl edit  machinedeployment <cluster-Namer-md>
 

3. The update the option with the new value under the  "annotations" section

 annotations:
    cluster.k8s.io/cluster-api-autoscaler-node-group-max-size: "10"
    cluster.k8s.io/cluster-api-autoscaler-node-group-min-size: "1"
 

4. Save the change and exit

Note: This will not trigger a "RollingUpdate" task as the "spec" section is not updated

 

• To update the following values, you will need to modify them under the cluster autoscaler deployment in the management cluster 

AUTOSCALER_MAX_NODES_TOTAL
AUTOSCALER_SCALE_DOWN_DELAY_AFTER_ADD
AUTOSCALER_SCALE_DOWN_DELAY_AFTER_DELETE AUTOSCALER_SCALE_DOWN_DELAY_AFTER_FAILURE
AUTOSCALER_SCALE_DOWN_UNNEEDED_TIME
AUTOSCALER_MAX_NODE_PROVISION_TIME

1. Run the following command to get the cluster deployment.

kubectl get deployment

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
cluster-one-cluster-autoscaler   1/1     1            1           16h

2. Run the following command to edit Edit the cluster deployment

kubectl edit deployment cluster-one-cluster-autoscaler

3. Modify the value you want under the containers -> args section

spec:
      containers:
      - args:
        - --cloud-provider=clusterapi
        - --v=4
        - --clusterapi-cloud-config-authoritative
        - --kubeconfig=/mnt/cluster-one-kubeconfig/value
        - --node-group-auto-discovery=clusterapi:clusterName=cluster-one
        - --scale-down-delay-after-add=10m
        - --scale-down-delay-after-delete=10s
        - --scale-down-delay-after-failure=3m
        - --scale-down-unneeded-time=10m
        - --max-node-provision-time=15m
        - --max-nodes-total=0

4. Save and exit , And this will delete the cluster autoscaler pod under the MGMT cluster and recreate a new one.

 Note:

• For more information on the autoscale option please refer to the documentation
• For each Tanzu Kubernetes cluster that you create with Autoscaler enabled, Tanzu Kubernetes Grid creates a Cluster Autoscaler deployment in the management cluster. To disable Cluster Autoscaler, delete the Cluster Autoscaler deployment associated with your Tanzu Kubernetes cluster. More details can be found in our documentation