How to identify the LEADER for Kubernetes Master endpoints in a Multi-Master Cluster configuration in PKS
Article ID: 298588
Updated On:
Products
Issue/Introduction
This article discusses how to identify the Leader for Kubernetes Master endpoints in a Multi-Master Cluster configuration within Pivotal Container Service (PKS). This is important for certain troubleshooting and analysis tasks, such as turning up debug levels on Kubernetes Master node components.
For certain troubleshooting and analysis tasks, you may want to determine the current active Leader of one or more Kubernetes components running within a Multi-Master HA configuration.
Environment
Resolution
For example, if you are planning to turn up the logging levels as described in How to turn up debug levels on Kubernetes Master node components in PKS, you will want to make sure to update the Leading Kubernetes component last.
This is because some Kubernetes components, such as kube-scheduler, require a quick restart to reflect the logging change. As a result, you will want any Leader to failover last.
Procedure: kube-scheduler
Note: This procedure requires BOSH credentials, which are available through your Opsmanager UI. You will also require kubernetes cluster admin credentials for access to the kube-system namespace.
1. Make sure you are logged in to the PKS API.
pks login -a <pks-api-address> -u <user> -k
2. Get Kubernetes credentials for your PKS multi-master cluster.
pks get-credentials your-cluster-name
3. Identify the universally unique identifier (UUID) of your PKS cluster, which will be be used later in this procedure.
export clusteruuid=$(pks cluster your-cluster-name --json | jq -r .uuid)
Example Output:
$ echo $clusteruuid ff395d2b-8830-4f9a-b28e-ea49dde4163e
4. For reference, get the details for the kube-scheduler endpoint of your PKS multi-master cluster:
kubectl get ep/kube-scheduler -n kube-system -o json
5. Filter the kube-scheduler endpoint for the current
MASTER_SCHEDULER_LEADER_AGENT_ID:
export MASTER_SCHEDULER_LEADER_AGENT_ID=$(kubectl get ep/kube-scheduler -n kube-system -o json | jq -r .metadata.annotations[] | awk -F',' '{print $1}' | awk -F':' '{print $2}' | jq -r . | awk -F'_' '{print $1}')
This is the BOSH Agent ID for the current Leader:
$ echo $MASTER_SCHEDULER_LEADER_AGENT_ID 7ca20473-6257-4058-b554-697e5d5ba6c1
6. For reference, list all Master node Instances:
bosh -d service-instance_${clusteruuid} is --details --column Instance --column 'Agent ID' --column Index --tty | egrep 'Instance|master'
Example output:
Instance Agent ID Index master/604657af-bffa-42ac-9756-1d52d5510370 7ca20473-6257-4058-b554-697e5d5ba6c1 2 master/97aa76e7-e4bc-49b7-9f18-4a6d24adea6f fa11a4ae-e728-447c-a08b-6720b512025f 0 master/ffc85cbc-1b0d-45f4-ab46-7781dbfc02ca 69cfe7b3-c83e-4aa9-bcdf-af1698700d42 1
7. Determine the current Master Leader instance using the MASTER_SCHEDULER_LEADER_AGENT_ID from above:
bosh -d service-instance_${clusteruuid} is --details --column Instance --column 'Agent ID' --column Index --tty | egrep --color=always $MASTER_SCHEDULER_LEADER_AGENT_ID
Example output:
master/604657af-bffa-42ac-9756-1d52d5510370 7ca20473-6257-4058-b554-697e5d5ba6c1 2
8. Use BOSH ssh to update the non-Leading instances first. In this case, the instances to update are index 0 and index 1.
bosh -d service-instance_${clusteruuid} ssh master/0
sudo -i
bosh -d service-instance_${clusteruuid} ssh master/1
sudo -i
Then you can follow the same steps in the article How to turn up debug levels on Kubernetes Master node components in PKS.
9. Update the last Leader instance logging, Index 2 in this case, so that any Leader failover will also have debug logging turned up:
bosh -d service-instance_${clusteruuid} ssh master/2
sudo -i
Procedure: kube-controller-manager
Note: This procedure requires BOSH credentials, which are available through your Opsmanager UI. You will also require kubernetes cluster admin credentials for access to the kube-system namespace.
1. Make sure you are logged in to the PKS API.
pks login -a <pks-api-address> -u <user> -k
2. Get Kubernetes credentials for your PKS multi-master cluster.
pks get-credentials your-cluster-name
3. Identify the UUID of your PKS cluster, which will be be used later in this procedure.
export clusteruuid=$(pks cluster your-cluster-name --json | jq -r .uuid)
Example output:
$ echo $clusteruuid
ff395d2b-8830-4f9a-b28e-ea49dde4163e
4. For reference, get the details for the kube-controller-manager endpoint of your PKS multi-master cluster:
kubectl get ep/kube-controller-manager -n kube-system -o json
5. Filter kube-controller-manager endpoint for the current MASTER_CONTROLLER_MANAGER_LEADER_AGENT_ID.
export MASTER_CONTROLLER_MANAGER_LEADER_AGENT_ID=$(kubectl get ep/kube-controller-manager -n kube-system -o json | jq -r .metadata.annotations[] | awk -F',' '{print $1}' | awk -F':' '{print $2}' | jq -r . | awk -F'_' '{print $1}')
This is the BOSH Agent ID for the current Leader:
$ echo $MASTER_CONTROLLER_MANAGER_LEADER_AGENT_ID
69cfe7b3-c83e-4aa9-bcdf-af1698700d42
6. For reference, list all Master node Instances:
bosh -d service-instance_${clusteruuid} is --details --column Instance --column 'Agent ID' --column Index --tty | egrep 'Instance|master'
Example output:
Instance Agent ID Index
master/604657af-bffa-42ac-9756-1d52d5510370 7ca20473-6257-4058-b554-697e5d5ba6c1 2
master/97aa76e7-e4bc-49b7-9f18-4a6d24adea6f fa11a4ae-e728-447c-a08b-6720b512025f 0
master/ffc85cbc-1b0d-45f4-ab46-7781dbfc02ca 69cfe7b3-c83e-4aa9-bcdf-af1698700d42 1
7. Determine the current Master Leader instance using the MASTER_SCHEDULER_LEADER_AGENT_ID from above:
bosh -d service-instance_${clusteruuid} is --details --column Instance --column 'Agent ID' --column Index --tty | egrep --color=always $MASTER_CONTROLLER_MANAGER_LEADER_AGENT_ID
Example output:
master/ffc85cbc-1b0d-45f4-ab46-7781dbfc02ca 69cfe7b3-c83e-4aa9-bcdf-af1698700d42 1
8. Use BOSH ssh to update the non-Leading instances first. In this case, the instances to update are index 0 and index 1.
bosh -d service-instance_${clusteruuid} ssh master/0
sudo -i
bosh -d service-instance_${clusteruuid} ssh master/2
sudo -i
Then you can follow the same steps in the article How to turn up debug levels on Kubernetes Master node components in PKS.
9. Update the last Leader instance logging, Index 2 in this case, so that any Leader failover will also have debug logging turned up:
bosh -d service-instance_${clusteruuid} ssh master/1
sudo -i
Checklist:
Procedure: kube-scheduler
Note: This procedure requires BOSH credentials, which are available through your Opsmanager UI. You will also require kubernetes cluster admin credentials for access to the kube-system namespace.
1. Make sure you are logged in to the PKS API.
pks login -a <pks-api-address> -u <user> -k
2. Get Kubernetes credentials for your PKS multi-master cluster.
pks get-credentials your-cluster-name
3. Identify the universally unique identifier (UUID) of your PKS cluster, which will be be used later in this procedure.
export clusteruuid=$(pks cluster your-cluster-name --json | jq -r .uuid)
Example Output:
$ echo $clusteruuid ff395d2b-8830-4f9a-b28e-ea49dde4163e
4. For reference, get the details for the kube-scheduler endpoint of your PKS multi-master cluster:
kubectl get ep/kube-scheduler -n kube-system -o json
5. Filter the kube-scheduler endpoint for the current MASTER_SCHEDULER_LEADER_AGENT_ID:
export MASTER_SCHEDULER_LEADER_AGENT_ID=$(kubectl get ep/kube-scheduler -n kube-system -o json | jq -r .metadata.annotations[] | awk -F',' '{print $1}' | awk -F':' '{print $2}' | jq -r . | awk -F'_' '{print $1}')
This is the BOSH Agent ID for the current Leader:
$ echo $MASTER_SCHEDULER_LEADER_AGENT_ID 7ca20473-6257-4058-b554-697e5d5ba6c1
6. For reference, list all Master node Instances:
bosh -d service-instance_${clusteruuid} is --details --column Instance --column 'Agent ID' --column Index --tty | egrep 'Instance|master'
Example output:
Instance Agent ID Index master/604657af-bffa-42ac-9756-1d52d5510370 7ca20473-6257-4058-b554-697e5d5ba6c1 2 master/97aa76e7-e4bc-49b7-9f18-4a6d24adea6f fa11a4ae-e728-447c-a08b-6720b512025f 0 master/ffc85cbc-1b0d-45f4-ab46-7781dbfc02ca 69cfe7b3-c83e-4aa9-bcdf-af1698700d42 1
7. Determine the current Master Leader instance using the MASTER_SCHEDULER_LEADER_AGENT_ID from above:
bosh -d service-instance_${clusteruuid} is --details --column Instance --column 'Agent ID' --column Index --tty | egrep --color=always $MASTER_SCHEDULER_LEADER_AGENT_ID
Example output:
master/604657af-bffa-42ac-9756-1d52d5510370 7ca20473-6257-4058-b554-697e5d5ba6c1 2
8. Use BOSH ssh to update the non-Leading instances first. In this case, the instances to update are index 0 and index 1.
bosh -d service-instance_${clusteruuid} ssh master/0
sudo -i
bosh -d service-instance_${clusteruuid} ssh master/1
sudo -i
Then you can follow the same steps in the article How to turn up debug levels on Kubernetes Master node components in PKS.
9. Update the last Leader instance logging, Index 2 in this case, so that any Leader failover will also have debug logging turned up:
bosh -d service-instance_${clusteruuid} ssh master/2
sudo -i
Procedure: kube-controller-manager
Note: This procedure requires BOSH credentials, which are available through your Opsmanager UI. You will also require kubernetes cluster admin credentials for access to the kube-system namespace.
1. Make sure you are logged in to the PKS API.
pks login -a <pks-api-address> -u <user> -k
2. Get Kubernetes credentials for your PKS multi-master cluster.
pks get-credentials your-cluster-name
3. Identify the UUID of your PKS cluster, which will be be used later in this procedure.
export clusteruuid=$(pks cluster your-cluster-name --json | jq -r .uuid)
Example output:
$ echo $clusteruuid ff395d2b-8830-4f9a-b28e-ea49dde4163e
4. For reference, get the details for the kube-controller-manager endpoint of your PKS multi-master cluster:
kubectl get ep/kube-controller-manager -n kube-system -o json
5. Filter kube-controller-manager endpoint for the current MASTER_CONTROLLER_MANAGER_LEADER_AGENT_ID.
export MASTER_CONTROLLER_MANAGER_LEADER_AGENT_ID=$(kubectl get ep/kube-controller-manager -n kube-system -o json | jq -r .metadata.annotations[] | awk -F',' '{print $1}' | awk -F':' '{print $2}' | jq -r . | awk -F'_' '{print $1}')
This is the BOSH Agent ID for the current Leader:
$ echo $MASTER_CONTROLLER_MANAGER_LEADER_AGENT_ID 69cfe7b3-c83e-4aa9-bcdf-af1698700d42
6. For reference, list all Master node Instances:
bosh -d service-instance_${clusteruuid} is --details --column Instance --column 'Agent ID' --column Index --tty | egrep 'Instance|master'
Example output:
Instance Agent ID Index master/604657af-bffa-42ac-9756-1d52d5510370 7ca20473-6257-4058-b554-697e5d5ba6c1 2 master/97aa76e7-e4bc-49b7-9f18-4a6d24adea6f fa11a4ae-e728-447c-a08b-6720b512025f 0 master/ffc85cbc-1b0d-45f4-ab46-7781dbfc02ca 69cfe7b3-c83e-4aa9-bcdf-af1698700d42 1
7. Determine the current Master Leader instance using the MASTER_SCHEDULER_LEADER_AGENT_ID from above:
bosh -d service-instance_${clusteruuid} is --details --column Instance --column 'Agent ID' --column Index --tty | egrep --color=always $MASTER_CONTROLLER_MANAGER_LEADER_AGENT_ID
Example output:
master/ffc85cbc-1b0d-45f4-ab46-7781dbfc02ca 69cfe7b3-c83e-4aa9-bcdf-af1698700d42 1
8. Use BOSH ssh to update the non-Leading instances first. In this case, the instances to update are index 0 and index 1.
bosh -d service-instance_${clusteruuid} ssh master/0
sudo -i
bosh -d service-instance_${clusteruuid} ssh master/2
sudo -i
Then you can follow the same steps in the article How to turn up debug levels on Kubernetes Master node components in PKS.
9. Update the last Leader instance logging, Index 2 in this case, so that any Leader failover will also have debug logging turned up:
bosh -d service-instance_${clusteruuid} ssh master/1
sudo -i
Additional Information
https://knowledge.broadcom.com/external/article?articleNumber=298557
Feedback
