DevTest Kubernetes Troubleshooting commands

Products

• Service Virtualization

Issue/Introduction

We have a devtest 10.X on-prem installation of Kubernetes.

What are some helpful commands to be able to troubleshoot the Kubernetes environment?

 

Environment

All supported DevTest releases and platforms.

Cause

N/A

Resolution

Please validate that your are using the correct version of HELM as helm being updated is one of the most frequent problems support sees.

Please see version information below:
DevTest Docker for Kubernetes Using Nodeports (broadcom.com)

Other useful commands.

Helm Commands

• Command to check the helm version
  1. Helm version
• Command to check all of the releases for a specified namespace
  1. helm list –namespace <helmnamespace>
• apply changes from custom-values.yaml
  1. helm upgrade <helmnamespace> ./devtest-0.X.X.tgz –install –values ./custom-values.yaml –namespace <namespace>
     Example:
     helm upgrade devtest107 ./devtest-0.1.7.tgz --install --values ./custom-values.yaml --namespace devtest-107
• helm upgrade command to use folder instead of tgz file
  1. helm upgrade <helmnamespace> ./devtest –install –values ./custom-values.yaml –namespace <namespace>
     Example:
     helm upgrade devtest107 ./devtest --install --values ./custom-values.yaml --namespace devtest-107
• remove helm deployment
  1. helm delete –purge <helmnamespace>
     Example:
     helm delete --purge  devtest107
• To restart pods this is only available before helm 3.X
  1. helm upgrade --recreate-pods <helmnamespace> ./devtest-0.1.7.tgz --install --values ./custom-values.yaml --namespace <namespace>
     Example:
     helm upgrade --recreate-pods devtest106 ./devtest-0.1.7.tgz --install --values ./custom-values.yaml --namespace devtest-106
• Command to update service account user
  1. helm init --service-account tiller --upgrade

Kubernetes Commands

• Command to apply new persistent volume information

kubectl apply –f pv.yaml

• Command to list pods on the cluster master

kubectl get po -n kube-system

• Command to list devtest pods in <namespace>

kubectl get pods -n <namespace>
Example:
kubectl get pods -n devtest-107

• Command to list devtest pods details in <namespace>

kubectl get pods -o wide -n <namespace>
Example:
kubectl get pods -o wide -n devtest-107

• Command to open command pod command prompt
  1. kubectl exec -it <podname> -n <namespace> -- /bin/sh

Example:
kubectl exec -it devtest107-portal-0 -n devtest-107 -- /bin/sh

NOTE: Logs are stored in /home/devtest/lisatmp_XX.XX

• Command to delete configmap for database driver and config files
  1. kubectl delete configmap <ConfigMapName> --namespace <namespace>

Example Database Driver:
kubectl delete configmap db-driver --namespace devtest-107
Example Config files::
kubectl delete configmap devtest-config --namespace devtest-107

• Command to create configmap for database driver and config files
  1. kubectl create configmap <ConfigMapName> --from-file=<fullpathtofile>-n <namespace>

Example Database Driver:
kubectl create configmap db-driver --from-file=./dbdriver/mysql-connector-java-5.1.45-bin.jar -n devtest-107
Example Config files:
kubectl create configmap devtest-config --from-file=./config -n devtest-107

• Command to check a pods logs for the last x number of hours
  1. kubectl logs --since=Xh <podname> -n <namespace>
     Example:
     kubectl logs --since=1h devtest107-registry-0 -n devtest-107
• Command to Taint the primary node to allow Tiller pod install
  1. kubectl taint nodes --all node-role.kubernetes.io/master-
• Command to remove Taint from the Cluster Master so devtest pods are not deployed there
  1. kubectl taint nodes $(hostname) node-role.kubernetes.io/master:NoSchedule
• Command to list kubernetes persistent volumes
  1. kubectl get pv
• Command to create service account
  1. kubectl create serviceaccount --namespace <anemspace> <serviceAccoutName>

Example:
kubectl create serviceaccount --namespace kube-system tiller

• Command to create role binding

kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller

• Command to deploy service account user

kubectl patch deploy --namespace kube-system tiller-deploy –p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'    

• Command to regenerate a join token for worker nodes:

kubeadm token create --print-join-command

• Command to list nodes in cluster

kubectl get nodes

• Command to get configuration information for a pod
  1. kubectl describe pod <poodname> -n <namespace>

Example:
kubectl describe pod devtest107-portal-0 -n devtest-107

• Command to copy folder from Pod.
  1. kubectl cp <some-namespace>/<some-pod>:/tmp/foo /tmp/bar
     Example:
     kubectl cp devtest-107/devtest107-registry-0:/home/devtest ./devtest
• Command to list ports
  1. kubectl get svc -o wide -n <namespace>
     Example:
     kubectl get svc -o wide -n devtest-107
• Command to list configmaps in a name space
  1. kubectl get configmap --namespace <namespace>
     Example:
     kubectl get configmap  --namespace devtest-107

• How to check the Image versions in your Kubernetes deployment:
  kubectl get pods -n <namespace> -o jsonpath="{.items[*].spec.containers[*].image}" |\
  tr -s '[[:space:]]' '\n' |\
  sort |\
  uniq -c

Example Command:
kubectl get pods -n devtest-106 -o jsonpath="{.items[*].spec.containers[*].image}" |\
tr -s '[[:space:]]' '\n' |\
sort |\
uniq -c

Example Output:
1 esd-sv-docker-virtual.artifactory-lvn.broadcom.net/sv/config-server:0.0.6.20
4 esd-sv-docker-virtual.artifactory-lvn.broadcom.net/sv/lisa:10.7.0.168
1 esd-sv-docker-virtual.artifactory-lvn.broadcom.net/sv/portal:10.7.0.70
1 esd-sv-docker-virtual.artifactory-lvn.broadcom.net/sv/virtual-service-catalog:1.7.6.42