Collecting vSAN support logs and uploading to Broadcom VCF Support
Article ID: 327035
Updated On:
Products
Issue/Introduction
This article provides information on how to collect VMware vSAN support logs and upload them to Broadcom VCF Support.
Environment
Resolution
vSAN support logs are contained in a normal ESXi support bundle in the form of vSAN traces. The vSAN support logs are gathered automatically by gathering the ESXi support bundle for the hosts.
- As vSAN is an object-network based storage system distributed across multiple ESXi hosts, when troubleshooting vSAN issues the entire cluster logs including vCenter are required for a complete analysis of the issue. For example:
- If you have a 5-host cluster using a RAID1 policy the 3 components that make up that object are distributed across 3 hosts in the cluster per vSAN's choosing. However, the error seen can be on a host that doesn't own any of those components or the object in question that needs to be troubleshot.
- If you have a 10-host cluster using a RAID5 or RAID6 policy the components that make up the object will be laid out across 4 or 6 hosts depending on the RAID 5/6 policy in use.
- So, as you can see based on the above examples if we don't have the entire cluster logs for analysis then we won't have all the necessary information to provide an accurate/complete analysis.
- The only exceptions when we wouldn't need the entire cluster logs is for a disk failure called out on a specific host or a PSOD where the backtrace doesn't mention vSAN, LSOM, and/or DOM. For these instances, just the affected host bundle will suffice.
- When troubleshooting vSAN Performance issues additional vSAN Performance data is required for analysis. See KB Collecting vSAN Performance Service data for vSAN performance issues for collecting this additional Perf data.
Collecting Diagnostic data from the vSAN Native Trace Object (vSAN 8.0 U1+)
If you are attempting to gather longer vSANTraces from the Native vSAN Trace Object introduced in vSAN 8.1 - please follow the following instructions and remember to check the overall size of logs before continuing as they may exceed 500GB if configured for long retention times:
1) Connect to the vSAN Host via SSH
2) Confirm that native traces cover the time frame of the issue by running the command bellow:
ls -R /vmfs/volumes/vsanDatastore/.vsan.trace/
3) If the traces cover the time frame run the following command to include all data from the vSAN Native Trace Object:
vm-support -a Storage:VSANTracesExtended -w <localdatastore path>
4) Append vSANtraces to the file name that was generated before upload.
5) Repeat steps 1-4 on all hosts in the vSAN cluster.
6) Provide the log file that results to VMware Support as you normally would
If you should receive the below warning when attempting to collect the native trace logs it means encryption is enabled on the cluster and you need to follow the steps provided to supply a password.
WARNING main.py:590 Command cannot succeed because this host is in crypto safe mode and the vm-support incident key is missing.
To collect useful coredumps, perform these tasks:
1. Generate a vm-support incident key by running:
crypto-util keys vm-support --password prolog
2. Run vm-support:
vm-support [options]
3. Perform cleanup:
crypto-util keys vm-support epilog- It's highly recommended that the cluster logs are collected before vCenter logs to ensure we get all the required logging from the hosts at the time of the event as host logs tend to wrap faster than vCenter logs.
- When dealing with large clusters 20+ hosts and collecting logs via vCenter it's best to collect the host logs in small batches, no more than 5 at a time to ensure the logs don't get corrupted.
Additional Information
Feedback
