Logs required to investigate vSphere Cloud Native Storage issues.
Article ID: 388188
Updated On:
Products
Issue/Introduction
To investigate issues with vSphere Cloud Native Storage logs from vCenter, ESXi hosts and the specific distribution of Kubernetes are required.
A Container Storage Plugin deployment in a kubernetes cluster consists of one 'vsphere-csi-controller' pod per master node with one 'vsphere-csi-node' pod per node within that cluster. There is one deployment per kubernetes cluster and one pod per worker node within that cluster.
The 'vsphere-csi-controller' pod consists of the following 7 containers:
csi-attacher, csi-resizer, vsphere-csi-controller, liveness-probe, vsphere-syncer, csi-provisioner, csi-snapshotter
While the 'vsphere-csi-node' pod on each worker consists of the following 3 containers:
node-driver-registrar, livenessprobe, vsphere-csi-node
Capturing the logs depends on the distribution of Kubernetes deployed.
This KB will cover
Vanilla Kubernetes
vSphere with Tanzu/vSphere Supervisor
Tanzu Kubernetes Grid Integrated
Environment
vSphere
vSphere with Tanzu
vSphere Supervisor
Resolution
vCenter logs contain information related to CNS regardless of the deployment of Kubernetes in use
CNS logs are captured via a VC log Bundle
Collecting diagnostic information for VMware vCenter Server 7.x and 8.x
If issues with collecting or providing logs the following are some of the most useful log files on the VC side relating to persistent volume problems which would provide a starting point for analyzing a CSI issue on the VC side:
/var/log/vmware/vsan-health/vsanvcmgmtd.log
/var/log/vmware/vsan-health/vmware-vsan-health-service.log (For any permissions or vSAN file system related issues)
/var/log/vmware/vmware-sps/sps.log
/var/log/vmware/vpxd/vpxd.log
The embedded vpostrgres database also contains valuable information
The following commands can be ran from an ssh session connected to the VC and the created file downloaded from /tmp.
- /opt/vmware/vpostgres/current/bin/pg_dump -U postgres -d VCDB -t VPX_STORAGE_OBJECT_INFO -t VPX_STORAGE_OBJECT_MD -t VPX_FCD_DATASTORE_INFO -t VPX_FCD_SNAPSHOT_INFO -t VPX_STORAGE_DEBUG_INFO > /tmp/FCDTables
2. /opt/vmware/vpostgres/current/bin/pg_dump -U postgres -d VCDB -n cns > /tmp/CNSTables
ESXi Host logs connected to the storage being used also helpful for specific investigation and can be captured following this KB
Collecting diagnostic information for ESXi hosts and vCenter Server using the vSphere Client
Vanilla Kubernetes
To capture the logs from these pods we first need to identify the namespace which they have been created in, we can do this by running the following command:
kubectl get pods -l role=vsphere-csi -A
Once we know the namespace we can use the following kubectl commands which use the pod label 'vsphere-csi-controller' and namespace e.g. 'vmware-system-csi' (default but possible to change) to identify the pods before outputting the logs to txt file:
kubectl logs -l app=vsphere-csi-controller -n vmware-system-csi -c csi-attacher > csi-attacher.log
kubectl logs -l app=vsphere-csi-controller -n vmware-system-csi -c csi-resizer > csi-resizer.log
kubectl logs -l app=vsphere-csi-controller -n vmware-system-csi -c vsphere-csi-controller > vsphere-csi-controller.log
kubectl logs -l app=vsphere-csi-controller -n vmware-system-csi -c liveness-probe > liveness-probe.log
kubectl logs -l app=vsphere-csi-controller -n vmware-system-csi -c vsphere-syncer > vsphere-syncer.log
kubectl logs -l app=vsphere-csi-controller -n vmware-system-csi -c csi-provisioner > csi-provisioner.log
kubectl logs -l app=vsphere-csi-controller -n vmware-system-csi -c csi-snapshotter > csi-snapshotter.log
To get the logs from the node pods the following one-liner should capture and output them into individual text files for each 'worker' node (puts logs from all 3 containers in each pod into a single text file):
for i in $(kubectl get pods -n "vmware-system-csi" -l "app=vsphere-csi-node" | sed 1d | cut -d" " -f1); do kubectl logs $i -n "vmware-system-csi" "--all-containers=true" > /tmp/$i.txt ;done
vSphere with Tanzu/vSphere Supervisor
Controller logs are captured with the full log bundle following the steps in this KB article – both Workload Management logs and Guest Cluster logs are required.
Gathering Logs for vSphere with Tanzu
Node logs are captured for the guest clusters only.
for i in $(kubectl get pods -n "vmware-system-csi" -l "app=vsphere-csi-node" | sed 1d | cut -d" " -f1); do kubectl logs $i -n "vmware-system-csi" "--all-containers=true" > /tmp/$i.txt ;done
Tanzu Kubernetes Grid Integrated
CSI Log Collection in TKGi
How to collect the logs for OpsManager Bosh director and system logs for Bosh VM managed
(Note this is for TKGi only and not all other TKG variants)
TKGi does not run the csi-controller as pods but as service on all Master and Worker nodes so the log collection process is different for it.
The exact path for CSI logs for a TKGi cluster is /var/vcap/sys/log/csi-controller for the masters and /var/vcap/sys/log/csi-node-service for the workers.
Tanzu Support should collect the logs from all Master and Worker nodes where the csi-controller is running as a service: from https://kb.vmware.com/s/article/91889 step 3:
DEPLOYMENT is the service-instance_ID for the TKGi cluster.
bosh -d DEPLOYMENT ssh master -c "sudo tar -czvf /var/vcap/sys/log/varlog.tgz /var/log && sudo tar -czvf /var/vcap/sys/log/varbosh.tgz /var/vcap/bosh/log"
bosh -d DEPLOYMENT logs master
bosh -d DEPLOYMENT ssh master -c "sudo rm /var/vcap/sys/log/varlog.tgz /var/vcap/sys/log/varbosh.tgz"
bosh -d DEPLOYMENT ssh worker -c "sudo tar -czvf /var/vcap/sys/log/varlog.tgz /var/log && sudo tar -czvf /var/vcap/sys/log/varbosh.tgz /var/vcap/bosh/log"
bosh -d DEPLOYMENT logs worker
bosh -d DEPLOYMENT ssh worker -c "sudo rm /var/vcap/sys/log/varlog.tgz /var/vcap/sys/log/varbosh.tgz"
Feedback
