Support Bundler(vks-support-bundler) Data Collection List

search cancel

Support Bundler(vks-support-bundler) Data Collection List

book

Article ID: 400451

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

According to KB345464, the vks-support-bundler tool gathers support bundles from vSphere Kubernetes Clusters (VKCs) to aid in troubleshooting Guest Cluster issues. To comply with internal security policies, some customers may need visibility into the data collected by this tool. This article outlines all data gathered by the vks-support-bundler for customer awareness.

Environment

vSphere with Tanzu

VMware vSphere Kubernetes Service

VKS Support Bundler

Resolution

Data collected from Linux node -

Logs:

Pod	/var/log/pods
Containers	/var/log/containers
Antrea	/var/log/antrea

Node status:

Disk usage	df &> df.out
Disk inode usage	df -i &> df-i.out
IP address	ip addr &> ip-addr.out
Socket summary (ss)	ss -anp &> df.out
Socket detailed (ss)	ss -aens &> ss-aens.out
Routing table	ip route &> ip-route.out
Mount points	mount &> mount.out
Kernel messages	dmesg &> dmesg.out
Free memory	free -h &> free-h.out
System uptime	uptime &> uptime.out
Current date	date &> date.out
Process list sorted by RSS	ps auwwx --sort -rss &> ps-auwwx.out
Ulimit	bash -c "ulimit -a" &> ulimit.out
Umask	bash -c "umask" &> umask.out
Memory info	cat /proc/meminfo &> meminfo.out
CPU info	cat /proc/cpuinfo &> cpuinfo.out
VM stats	cat /proc/vmstat &> vmstat.out
Swap info	cat /proc/swaps &> swaps.out
Mounts info	cat /proc/mounts &> mounts.out
IP forward	cat /proc/sys/net/ipv4/ip_forward &> ip-forward.out
Time sync config	cat /etc/systemd/timesyncd.conf &> timesyncd.out
IP neighbors	ip neigh &> ip-neigh.out
Environment variables	env &> env.out
Top command	top -d 5 -n 5 -b &> top.out
Iptables rules	sudo iptables -L -n &> iptables.out
Iptables save	sudo iptables-save &> iptbles-save.out
IP link show	ip link show &> ip-link.out
Docker status	systemctl status docker &> docker.out
crictl ps	sudo crictl ps -a &> ps-a.out
crictl pods	sudo crictl pods &> pods.out
crictl info	sudo crictl info &> info.out
Disk usage for containerd	df -h /var/lib/containerd &> containerd.out
crictl images	sudo crictl images &> images.out
Load GC containers log	sudo journalctl -u load-gc-containers &> load-gc-containers.out
Kubelet status	systemctl status kubelet &> kubelet.out
Kubelet journal logs	sudo journalctl -xeu kubelet &> journalctl-kubelet.out
Containerd journal logs	sudo journalctl -xeu containerd &> journalctl-containerd.out
Cloud-init output log	sudo cat /var/log/cloud-init-output.log &> cloud-init-output.out
Cloud-init log	sudo cat /var/log/cloud-init.log &> cloud-init.out
Open internet sockets	lsof -i -n &> lsof.out
File descriptor stats	sysctl fs.file-nr &> fs-file-nr.out
DNS resolvers	cat /etc/resolv.conf &> resolv-conf.out
Swap on status	swapon -s &> swapon.out
nf_conntrack max	cat /proc/sys/net/netfilter/nf_conntrack_max &> nf-conntrack-max.out
File max	cat /proc/sys/fs/file-max &> file-max.out

Ulimit configuration	ulimit -a &>ulimit.out
System parameters (sysctl)	sysctl -a &>sysctl.out
System load average	cat /proc/loadavg &>loadavg.out
Maximum number of threads	cat /proc/sys/kernel/threads-max &>threads-max.out
Full process list	ps -elf &>pself.out
Process and thread list	ps -elfT &>psthreads.out
Open file stats (current)	cat /proc/sys/fs/file-nr &>file-nr.out
Max open file limit	cat /proc/sys/fs/file-max &>file-max.out
List of open files	lsof -n &>lsof-n.out
Kernel and system info	uname -a &>uname.out
Virtual memory stats	vmstat 1 5 &>vmstat.out
All system activity report	sar -A &>sar-A.out
Detailed process stats	pidstat -p ALL -T ALL -I -l -r -t -u &>pidstat.out
Virtual memory page stats	cat /proc/vmstat &>proc-vmstat.out
Top memory-consuming processes	ps -eo pid,ppid,cmd,vsz,fuser,%mem,%cpu --sort=-%mem \| head -n 20 &>psmem.out
Memory info	cat /proc/meminfo &>mem-info.out
Memory slab usage	vmstat -m &>vmstat-m.out
Slab allocator info	cat /proc/slabinfo &>slabinfo.out
Slab memory usage (live)	slabtop -s c -o &>slabtop.out
Memory usage summary	free -m &>free-m.out
Top CPU-consuming processes	ps -eo pid,ppid,cmd,vsz,fuser,%mem,%cpu --sort=-%cpu \| head -n 20 &>pscpu.out
CPU information	cat /proc/cpuinfo &>cpuinfo.out
SoftIRQ stats	cat /proc/softirqs &>softirqs.out
Interrupt stats	cat /proc/interrupts &>interrupts.out
CPU topology	lscpu &>lscpu.out
System uptime	uptime &>uptime.out
Per-core CPU usage	mpstat 1 5 -P ALL &>mpstat-all.out
Network connections summary	netstat -lantupWe &>netstat-lantupWe.out
Interface-level stats	netstat -i &>netstat-i.out
Network protocol stats	netstat -s &>netstat-s.out
Socket states and endpoints	ss -noemitaup &>ss-noemitaup.out
Network device stats	cat /proc/net/dev &>proc-net-dev.out
IPv4 socket stats	cat /proc/net/sockstat &>sockstat.out
IPv6 socket stats	cat /proc/net/sockstat6 &>sockstat6.out
IP address configuration	ip addr &>ip-addr.out
Route table	route -n &>route-n.out
IP rule list	ip rule list &>ip-rule-list.out
All route tables	ip route show table all &>ip-show-table.out
Iptables rules	iptables -L -v -n --line-numbers &>iptables-all.out
Connection tracking table	conntrack -L &>conntrack-L.out
Conntrack statistics	conntrack -S &>conntrack-S.out
Disk usage (inodes)	df -ih --total &>df-ih.out
Disk usage (all files)	df -ah --total &>df-ah.out
ARP cache	arp -an &>arp-an.out
I/O stats	iostat 1 5 &>iostat.out

Etcd status:

ETCD member list	sudo crictl exec -i $PID etcdctl ${ETCD_CERT_OPTION} member list --write-out=table >f
ETCD endpoint status	sudo crictl exec -i $PID etcdctl ${ETCD_CERT_OPTION} --cluster=true endpoint status --write-out=table >endpoint_status.log
ETCD endpoint health	sudo crictl exec -i $PID etcdctl ${ETCD_CERT_OPTION} --cluster=true endpoint health --write-out=table >f

System namespace data collection -

This script collects diagnostic data from key Kubernetes system namespaces by dumping cluster information and retrieving resource details such as PersistentVolumeClaims, Endpoints, ConfigMaps, CronJobs, etc.

Namespaces involved:

kube-system

tkg-system

secretgen-controller

vmware-system-auth

vmware-system-cloud-provider

vmware-system-csi

vmware-system-tkg

Resources collected for each namespace:

persistentvolumeclaims

endpoints

configmaps

cronjobs

app

pkgi

pdb

dumpinfo(kubectl cluster-info dump -n ns)

Data collected from Windows node -

Logs:

Pod	C:\var\log\pods
Container Registry	C:\var\log\containers
Antrea	C:\var\log\antrea
Containers	C:\var\log\containers
Kubelet	C:\var\log\kubelet
Could Base Init	C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log
First Boot(Out of Box Experience)	C:\Windows\Panther\setupact.log C:\Windows\Panther\setuperr.log

NodeStatus:

Name	Details
Network Info	Get-NetIpAddress \| convertto-json Get-NetAdapter \| convertto-json Get-NetRoute \| convertto-json See TCP connection script See UDP connection script
CpuInfo	Get-WmiObject Win32_Processor \| convertto-json Get-Counter \| convertto-json
DiskDiskInfo	Get-PSDrive -PSProvider FileSystem Get-Volume \| convertto-json get-partition \| convertto-json
IOInfo	Get-Counter -Counter "\LogicalDisk(_Total)\*" \| convertto-json
NetworkIOInfo	Get-Counter -Counter "\Network Interface()\" \| convertto-json
MemoryInfo	Get-WmiObject Win32_OperatingSystem \| convertto-json Get-Counter -Counter "\Memory\*" \| convertto-json
ProcessInfo	See Process Information Script
EventLog(selected, default collection for 12 hours)	get-winevent -logname system \| convertto-json -depth 10 Get-WinEvent -LogName “Security” \| convertto-json -depth 10 Get-WinEvent -LogName “Application” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-WinInet-Config/ProxyConfigChanged” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Windows Firewall” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Windows Defender” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-VolumeSnapshot” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Time” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Storage” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Security” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Partition” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Hyper-V” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Host-Network” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Containers” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Client-Licensing” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Dhcp” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Kernel-IO” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Kernel-Pnp” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Kernel-Cache” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Kernel-Boot” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Ntfs” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-WinInet-Config/ProxyConfigChanged” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Windows Firewall” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Windows Defender” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-VolumeSnapshot” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Time” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Storage” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Security” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Partition” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Hyper-V” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Host-Network” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Containers” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Client-Licensing” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Dhcp” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Kernel-IO” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Kernel-Pnp” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Kernel-Cache” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Kernel-Boot” \| convertto-json -depth 10 Get-WinEvent -LogName “Microsoft-Windows-Ntfs” \| convertto-json -depth 10 (note that by default time within three days is collected)
Containerd metrics (not enabled yet)	(invoke-webrequest -usebasicparsing http://localhost:9113/metrics).Content
Service settings	Get-Item HKLM:\SYSTEM\CurrentControlSet\Servicesegistry\ Get-Item HKLM:\SYSTEM\CurrentControlSet\Services\kubelet\ Get-Item HKLM:\SYSTEM\CurrentControlSet\Services\containerd\ Get-childitem -recurse HKLM:\SYSTEM\CurrentControlSet\Servicesegistry Get-childitem -recurse HKLM:\SYSTEM\CurrentControlSet\Services\kubelet Get-childitem -recurse HKLM:\SYSTEM\CurrentControlSet\Services\containerd

Process Information:

Get-Process | foreach-object {
 $threadInfo = foreach ($thread in $_.Threads) {
   @{
     "Id"         = $thread.Id;
     "Priority"   = @{
       "BasePriority"         = $thread.BasePriority;
       "CurrentPriority"      = $thread.CurrentPriority;
       "PriorityClass"        = $thread.PriorityClass;
       "PriorityBoostEnabled" = $thread.PriorityBoostEnabled;
       "PriorityLevel"        = $thread.PriorityLevel;
     };
     "Processor"  = @{
       "TotalTime"      = $thread.TotalProcessorTime;
       "UserTime"       = $thread.UserProcessorTime;
       "PrivilegedTime" = $thread.PrivilegedProcessorTime;
     };
     "State"      = $thread.ThreadState;
     "StartTime"  = $thread.StartTime.DateTime;      
     "WaitReason" = $thread.WaitReason;
   }
 }

$moduleInfo = foreach ($module in $_.Modules) {
   @{
     "Name"        = $module.ModuleName;
     "Filename"    = $module.Filename;
     "Memory"      = @{
       "BaseAddress"       = $module.BaseAddress;
       "ModuleMemorySize"  = $module.ModuleMemorySize;
       "EntryPointAddress" = $module.EntryPointAddress;
       "Size"              = $module.Size;
     };
     "FileVersion" = $module.FileVersion;
   }
 }

 @{
   "Id"          = $_.Id;
   "ProcessName" = $_.ProcessName;
   "Name"        = $_.Name;
   "Handle"      = $_.Handle; 
   "SessionId" = $_.SessionId;
     "Responding"  = $_.Responding;
   "Priority"    = @{
     "BasePriority"         = $_.BasePriority;
     "PriorityClass"        = $_.PriorityClass;
     "PriorityBoostEnabled" = $_.PriorityBoostEnabled;
   };
   "StartInfo"   = @{
     "Verb"             = $_.StartInfo.Verb;
     "Arguments"        = $_.StartInfo.Arguments;
     "CreateNoWindow"   = $_.StartInfo.CreateNoWindow;
     "UseShellExecute"  = $_.StartInfo.UseShellExecute;
     "UserName"         = $_.StartInfo.UserName;
     "Domain"           = $_.StartInfo.Domain;
     "LoadUserProfile"  = $_.StartInfo.LoadUserProfile;
     "WorkingDirectory" = $_.StartInfo.WorkingDirectory;
   };
   "StartTime"   = $_.StartTime.DateTime;
   "Processor"   = @{
     "TotalTime"         = $_.TotalProcessorTime;
     "UserTime"          = $_.UserProcessorTime;
     "PrivilegedTime"    = $_.PrivilegedProcessorTime;
     "CurrentPercentage" = $_.CPU;
     "Affinity"          = $_.ProcessorAffinity;
   };
   "Memory"      = @{
     "VirtualMemorySize"          = $_.VirtualMemorySize;
     "VirtualMemorySize64"        = $_.VirtualMemorySize64;
     "WorkingSet"                 = $_.WorkingSet;
     "WorkingSet64"               = $_.WorkingSet64;
     "NonpagedSystemMemorySize"   = $_.NonpagedSystemMemorySize;
     "NonpagedSystemMemorySize64" = $_.NonpagedSystemMemorySize64;
     "PagedMemorySize"            = $_.PagedMemorySize;
     "PagedMemorySize64"          = $_.PagedMemorySize64;
     "PagedSystemMemorySize"      = $_.PagedSystemMemorySize;
     "PagedSystemMemorySize64"    = $_.PagedSystemMemorySize64;
     "PeakPagedMemorySize"        = $_.PeakPagedMemorySize;
     "PeakPagedMemorySize64"      = $_.PeakPagedMemorySize64;
     "PeakWorkingSet"             = $_.PeakWorkingSet;
     "PeakWorkingSet64"           = $_.PeakWorkingSet64;
     "PeakVirtualMemorySize"      = $_.PeakVirtualMemorySize;
     "PeakVirtualMemorySize64"    = $_.PeakVirtualMemorySize64;
     "PrivateMemorySize"          = $_.PrivateMemorySize;
     "PrivateMemorySize64"        = $_.PrivateMemorySize64;
     "MaxWorkingSet"              = $_.MaxWorkingSet;
       "MinWorkingSet"              = $_.MinWorkingSet;
     "BaseAddress"                = $_.MainModule.BaseAddress;
     "MainModuleMemorySize"       = $_.MainModule.ModuleMemorySize;
     "EntryPointAddress"          = $_.MainModule.EntryPointAddress;
   };
   "HandleCount" = $_.HandleCount;
   "Threads"     = $threadInfo;
   "FileInfo"    = @{
     "Filename"       = $_.MainModule.Filename;
     "VersionInfo"    = $_.MainModule.FileVersionInfo;
     "Version"        = $_.FileVersion;
     "Company"        = $_.Company;
     "ProductVersion" = $_.ProductVersion;
     "Description"    = $_.Description;
     "Product"        = $_.Product;
   };
   "Modules"     = $moduleInfo;
 }
} | convertto-json -Depth 10

TCP Connections:

Get-NetTCPConnection | foreach-object {
 $process = Get-Process -Id $_.OwningProcess

 @{
   "LocalAddress"      = $_.LocalAddress;
   "LocalPort"         = $_.LocalPort;
   "RemoteAddress"     = $_.RemoteAddress;
   "RemotePort"        = $_.RemotePort;
   "OffloadState"      = $_.OffloadState.ToString();
   "OwningProcess"     = $_.OwningProcess;
   "OwningProcessName" = $process.ProcessName;
   "OwningProcessPath" = $process.Path;
   "State"             = $_.State.ToString();
   "CreationTime"      = $_.CreationTime.DateTime;
 }
} | convertto-json -depth 10

UDP Connections:

Get-NetUDPEndpoint | foreach-object {
 $process = Get-Process -Id $_.OwningProcess
 @{
   "LocalAddress"      = $_.LocalAddress;
   "LocalPort"         = $_.LocalPort;
   "OwningProcess"     = $_.OwningProcess;
   "OwningProcessName" = $process.ProcessName;
   "OwningProcessPath" = $process.Path;
   "CreationTime"      = $_.CreationTime.DateTime;
 }
} | convertto-json -depth 10

Additional Information

Starting with vks-support-bundler v3.5.0, the bundle will also include:

Additional system data to aid in diagnosing slow boot issues
Output from kubeadm certs check-expiration to verify certificate expiration times
System logs related to systemd-networkd for improved network diagnostics

systemd-analyze blame	systemd-analyze blame > systemd-blame.txt
systemd-analyze dump	systemd-analyze dump > systemd-dump.txt
certs check-expiration	kubeadm certs check-expiration > kubeadm-certs-check-expiration.out
systemd-networkd	sudo journalctl -u systemd-networkd > systemd-networkd.log

- Please check https://knowledge.broadcom.com/external/article/345464 to know more about the vks-support-bundler and download the binary.

Feedback

thumb_up Yes

thumb_down No