Collecting ITCM Related Crash Dump Files
Article ID: 54925
Updated On:
Products
Issue/Introduction
Description:
Sometimes a problem is sufficiently complex or vague in cause that the products internal tracing is not sufficient and we need to capture crash and dump files on the OS level. This is the proper procedure to do that.
Solution:
NOTE: Along with the diagnostics information generated using this document it is very important that DSMInfo is also collected while the logs were set in DETAIL. The tracing for ITCM can be set to DETAIL using the following command:
cftrace -c set -l DETAIL -s 40000 -ln 6
Common items to trace and the tools available for each.
I. Process Crash
The most important diagnostic information for a process crash is dump of the crash.
To collect a dump of a crashing process, various tools/utilities can be used. We recommend using/trying the following tools in the order given below.
1.1 Userdump
Userdump can be used to generate a dump for a crashing process.
For detailed information on obtaining and installing this tool, visit the following link:
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=E089CA41-6A87-40C8-BF69-28AC08570B7E&displaylang=en
Once userdump is installed it can be configured to create a dump for the crashing process using the following steps.
- Go to Control Panel and click on "Process Dumper"
- Click on "New" and enter the name of the process which is crashing and click on OK.
- Click on "Rules" and select "Use custom rules" and select the appropriate "Exception Codes". For majority of the issues which we see "Access violation (c0000005)" and this should be selected unless you are explicity told otherwise by a support engineer. NOTE: This is the CRITICAL PART!
- Change the location of "Dump File Folder" if you want the dumps to be generated in a specific directory.
- Set "MiniDump Type" to "Complete". Once everything is configured click on OK.
- Click on "Apply" and then "OK" in the main window.
Now userdump is monitoring the process for any exceptions. As soon as an exception is generated a dump file will be created in the directory specified in "Dump File Folder". Send us this dump file.
For detailed information regarding using userdump, visit the following link:
http://support.microsoft.com/kb/241215
There are however, many type of crashes and tools that can be used to collect the relevant info. Here are several more scenarios and tools:
1.2 Dr Watson
Dr. Watson is a legacy tool supplied with WinXP and earlier versions of Windows which can be used to generate dump for crashing processes.
To configure Dr. Watson to create dumps use the following steps.
- Click Start, and then click Run.
- Type drwtsn32, and then click OK.
- Set the 'Crash Dump Type' to Full.
- Select the following 'Options'.
- Dump symbol table
- Dump all thread contexts
- Append to existing log files
- Create crash dump file
- Dump symbol table
By default the dump is created in the file:
C:\Documents and Settings\All Users\Application Data\Microsoft\Dr Watson\user.dmp
This can be changed if you want to create the dump in a different folder. Dr Watson will overwrite the dump file if another crash occurs. So it's better if the user.dump file is backed after each crash. Send us the dump file.
For detailed information on using Dr Watson, visit the following link:
http://support.microsoft.com/kb/275481
1.3 DebugDiag
DebugDiag tool can be used to generate memory dumps for process crash.
Use the steps mentioned below to install and configure DebugDiag:
- Install DebugDiag from http://www.microsoft.com/downloads/en/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en
- Start DebugDiag and select "Crash" in the "Select Rule Type" dialog and click "Next".
- In the "Select Target Type" dialog select "A specific process" and click on "Next".
- In the "Select Target" dialog select the process which is crashing and select "Next".
- In the "Advanced Configuration (Optional)" dialog click on "Exceptions...".
- In the "First Chance Exception Configuration" dialog click on "Add Exception...".
- In the "Configure Exception" dialog select the appropriate exception type. For majority of the issues which we see "C0000005 Access Violation" should be selected. Once the exception type is selected click on "OK"
- In the "First Chance Exception Configuration" dialog click on "Save & Close".
- In the "Advanced Configuration (Optional)" dialog click on "Next".
- In the "Select Dump Location and Rule Name (Optional)" window specify the rule name if you don't link the default rule name which is specified. The "Userdump Location" can also be specified if you want the dumps to be created in a specific folder. Click on "Next" once this is configured.
- In the "Rule Completed" dialog select "Activate the rule now" and select "Finish".
- Once the process crashes a memory dump will be created in the folder specified in step 10. Send us this dump file.
Note: If the process is crashing on x64 (64 bit machine) you still need to use x86 version of debugDiag since the process to be debugged is 32 bit.
1.4 AdPlus
AdPlus is a VB script supplied with "Debugging Tools for Windows". Adplus can be used in crash mode to generate a dump for a crashing process.
To get AdPlus you will have to install "Debugging Tools for Windows" from the following link:
http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx#b
The current release of "Debugging Tools for Windows" is included in the "Windows SDK". To download "Windows SDK" go to the following link:
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=6b6c21d2-2006-4afa-9702-529fa782d63b&displaylang=en
In the installation of "Windows SDK", during the "Installation Options" dialog ensure that you select only "Debugging Tools for Windows" under "Common Utilities".
By default the "Debugging Tools for Windows" will get installed in "C:\Program Files\Debugging Tools for Windows (x86)"
To generate a dump of a crashing process use the following steps:
- Open a command prompt and cd to "C:\Program Files\Debugging Tools for Windows (x86)".
- Execute the command "AdPlus -crash -pn <ProcessName> -o <dirForDump>". For example if you want to generate a dump for cmEngine in "C:\dump_dir" then execute "AdPlus -crash -pn cmEngine.exe -o C:\dump_dir".
- A dump file will be created by AdPlus within a folder in the specified path. Send us this dump file.
For detailed information regarding AdPlus, visit the following link:
http://support.microsoft.com/kb/286350/
II. Hung Process
Just like process crash the most important diagnostic information for a process hang is dump of the hanging process.
To collect a dump of a hanging process various tools/utilities can be used. We recommend using/trying the following tools in the order given below.
2.1 Userdump
Userdump can be used to generate a dump for a hanging process.
For detailed information on obtaining and installing this tool, visit the following link:
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=E089CA41-6A87-40C8-BF69-28AC08570B7E&displaylang=en
Once userdump is installed it can be configured to create a dump for the hanging process using the following steps.
- Find out the PID (Process ID) of the process which is hanging. To obtain the PID of the program, open Task Manager, and click on "Process" tab.
- Open a command prompt and cd to the directory "C:\kktools\userdump8.1\x86"
- Execute the command "userdump <PID>". For example if the PID of the hanging process was 2356 then execute "userdump 2356".
- A <processName>.dmp file will be created by userdump. Send us this dump file.
For detailed information regarding using userdump, visit the following link:
http://support.microsoft.com/kb/241215
2.2 AdPlus
AdPlus is a VB script supplied with "Debugging Tools for Windows". Adplus can be used in hang mode to generate a dump for a hanging process.
To get AdPlus you will have to install "Debugging Tools for Windows" from the following link:
http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx#b
The current release of "Debugging Tools for Windows" is included in the "Windows SDK". To download "Windows SDK" go to the following link:
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=6b6c21d2-2006-4afa-9702-529fa782d63b&displaylang=en
In the installation of "Windows SDK", during the "Installation Options" dialog ensure that you select only "Debugging Tools for Windows" under "Common Utilities".
By default the "Debugging Tools for Windows" will get installed in "C:\Program Files\Debugging Tools for Windows (x86)"
To generate a dump of a hanging process, use the following steps:
- Find out the PID (Process ID) of the process which is hanging. To obtain the PID of the program, open Task Manager, and click on "Process" tab.
- Open a command prompt and cd to "C:\Program Files\Debugging Tools for Windows (x86)".
- Execute the command "AdPlus -hang -p <PID> -o <dirForDump>". For example if the PID of the hanging process was 2356 and you want to generate the dumps in "C:\dump_dir" then execute "AdPlus -hang -p 2356 -o C:\dump_dir".
- A dump file will be created by AdPlus within a folder in the specified path. Send us this dump file.
For detailed information regarding AdPlus, visit the following link:
http://support.microsoft.com/kb/286350/
III. Process Memory Leak
To debug memory leak problems we can use various tools/utilities. We recommend using/trying the following tools in the order given below.
Before we use various tools/utilities to debug memory leak problems, it is required to set/enable the "Create user mode stack trace database" flag for the leaking process. This flag can be set using the following command:
gflags -i <processName> +ust
For example if you want to set the flag for cmEngine.exe then the command would be:
gflags -i cmEngine.exe +ust
Once this flag is set it is necessary to restart the process so that the new setting is picked up by the operating system.
3.1 DebugDiag
DebugDiag tool can be used to generate memory dumps for memory and handle leak.
Use the steps mentioned below to install and configure DebugDiag:
- Install DebugDiag from http://www.microsoft.com/downloads/en/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en
- Start DebugDiag and select "Memory and Handle Leak" in the "Select Rule Type" dialog and click "Next".
- In the "Select Target" dialog select the process which is leaking memory and select "Next".
- In the "Configure Leak Rule" dialog click on "Configure..." for "Userdump generation".
- In the "Configure userdumps for Leak Rule" dialog remove the check box for "Auto-create a crash rule to get userdump on unexpected process exit" and select "Save & Close".
- Click "Next" in the "Configure Leak Rule" dialog.
- In the "Select Dump Location and Rule Name (Optional)" window specify the rule name if you don't link the default rule name which is specified. The "Userdump Location" can also be specified if you want the dumps to be created in a specific folder. Click on "Next" once this is configured.
- In the "Rule Completed" dialog ensure that "Activate the rule now" is selected and then click on "Finish".
- Reproduce the leak or wait for the process to leak some memory. Normally you should wait for at least an hour (unless the leak occurs really quickly).
- Right click on "Leak rule for <processname>" and select "Dump Target Processes" -> "Create Full Userdump".
- This will create a dump file in the folder specified in step 7. Send us this dump file.
Note: If the process is leaking on x64 (64 bit machines) you still need to use x86 version of debugDiag since the process to be debugged is 32 bit.
3.2 UMDH
UMDH (User Mode Dump Heap) is supplied with "Debugging Tools for Windows". UMDH can be used to find out memory leaks in a process.
To get UMDH you will have to install "Debugging Tools for Windows" from the following link:
http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx#b
The current release of "Debugging Tools for Windows" is included in the "Windows SDK". To download "Windows SDK" goes to the following link:
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=6b6c21d2-2006-4afa-9702-529fa782d63b&displaylang=en
In the installation of "Windows SDK", during the "Installation Options" dialog ensure that you select only "Debugging Tools for Windows" under "Common Utilities".
By default the "Debugging Tools for Windows" will get installed in "C:\Program Files\Debugging Tools for Windows (x86)"
Use the steps mentioned below to generate data using UMDH:
- Find out the PID (Process ID) of the process which is leaking memory. To obtain the PID of the program, open Task Manager, and click on "Process" tab.
- Open a command prompt and cd to "C:\Program Files\Debugging Tools for Windows (x86)".
- Execute the command "umdh -p :<PID> -f:<fileName>". For example if the PID of the leaking process is 2356 and you want to dump the heap information in one.txt use the command "umdh -p:2356 -f:one.txt".
- This will dump the heap allocation information in one.txt
- Now reproduce the leak (if the steps are known) or wait for the memory usage of the leaking memory to increase.
- Execute the command specified in step 3 again but this time specifies a new file name. For example if the PID of the leaking process if 2356 and you want to dump the heap information in two.txt use the command "umdh -p:2356 -f:two.txt".
- This will dump the heap allocation information in two.txt.
- Send us the 2 files created using step 3 and step 6.
For detailed information regarding UMDH, visit the following link:
http://msdn.microsoft.com/en-us/library/ff560206(VS.85).aspx
As always, it is best to confirm all details with support before implementing. Many scenarios require customizations to several of these processes and they will not always be completely useful as is.
Environment
Component:
Feedback
