How To Enable vSphere CSI Controller Development/Debug logging on TKG Package Managed CSI

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

Outline a process for enabling debug logging in vSphere CSI Controller on TKG clusters that are managed TKG packages for troubleshooting purposes. In some troubleshooting situations you will need to see development/debug logs for CNS or CSI errors, this KB will assist in doing so.

Resolution

Follow the next steps to stop the CSI package from reconciling and overriding any changes made on the CSI controller yaml:

1. Change context to the TKG workload cluster that you would like to enable development logging on.
2. Run the following command to collect the package name, namespace, and status of the CSI package deployment:
   • kubectl get pkgi -A |grep -i csi
3. Pause the CSI controller package reconcile with the following command and the append name and namespace that you collected in the previous step:
   • kubectl patch pkgi/<PACKAGE_NAME> -n <PACKAGE_NAMESPACE> -p '{"spec":{"paused":true}}' --type=merge

Next follow these steps to set the CSI Controller logging to development/debug logging:

1. In the TKG workload cluster context run the following command to collect the CSI deployment name and namespace:
   • kubectl get deployments -A | grep -i csi
2. Edit the deployment of the CSI controller with the following command and append the name and namespace that you collected in the previous step:
   • kubectl edit deployment -n <CSI_CONTROLLER_NAMESPACE> <CSI_CONTROLLER_DEPLOYMENT_NAME>
3. Once in the edit view of the deployment under the name: LOGGER_LEVEL  there will be a value for "PRODUCTION" find and replace it with "DEVELOPMENT" in all places. Then save the edits on the file. After setting the values the CSI controller pods will restart and the logs for the controller will now have DEBUG output.

After collecting the needed logs for troubleshooting you can unpause the reconcile of the package and all edits on the package will be reconciled and overwritten to the default after approximately 5 - 10 min. Follow these commands to unpause the reconcile:

1. Change context to the TKG workload cluster that you enabled development logging on.
2. Run the following command to collect the package name and namespace of the CSI package deployment:
   • kubectl get pkgi -A |grep -i csi
3. Unpause the CSI controller package reconcile with the following command and the appended name and namespace that you collected in the previous step:
   • kubectl patch pkgi/<PACKAGE_NAME> -n <PACKAGE_NAMESPACE> -p '{"spec":{"paused":false}}' --type=merge

Additional Information

Example yaml file of were to make the DEVELOPMENT/PRODUCTION edits: https://github.com/kubernetes-sigs/vsphere-csi-driver/blob/5e42551faf8b61e3f4c64e5aa1f1eb3a08c8ac2d/manifests/vanilla/vsphere-csi-driver.yaml#L298