Troubleshooting Native Client (NC) Crash Dump Files with WinDbg

Products

• VMware Tanzu Gemfire

Issue/Introduction

Symptoms:
This article describes how to install, configure and use DebugDiag to troubleshoot crash dump files generated by the Native Client.

Environment

Resolution

WinDbg is a tool created by Microsoft. It works in MS Visual Studio. 

Install and configure WinDbg

Install Microsoft Visual Studio and then the WDK that includes WinDbg from this link
Open WinDbg

Set Symbol File Path from the menu: go to File -> Symbol File Path, paste the following path and push [OK]:

SRV*c:\winsym*http://msdl.microsoft.com/download/symbols;\\path\to\NativeClient\bin

Note that the path: \\path\to\NativeClient\bin depends on your environment according to install directory of the target version of Native Client

Analyse the crash dump file

Open dmp file from the menu: go to File -> Open Crash Dump

You can see the console and execute the command:!analyze -v at the bottom of the console window.

It will take several minutes to finish and will show the results afterward as shown in the following:

*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************

*** WARNING: Unable to verify timestamp for customLL.Common.dll
*** ERROR: Module load completed but symbols could not be loaded for customLL.Common.dll
*** WARNING: Unable to verify timestamp for UMI.CommonImplcustomLL.dll
*** ERROR: Module load completed but symbols could not be loaded for UMI.CommonImplcustomLL.dll
*** WARNING: Unable to verify timestamp for UMI.CommonImpl.dll
*** ERROR: Module load completed but symbols could not be loaded for UMI.CommonImpl.dll
Unable to load image C:\WINNT\assembly\NativeImages_v4.0.30319_32\mscorlib\3f95a6d480ed1ebe45cf27b770ba94ed\mscorlib.ni.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for mscorlib.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for mscorlib.ni.dll
Unable to load image c:\WINNT\Microsoft.NET\Framework\v4.0.30319\mscoreei.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for mscoreei.dll
*** WARNING: Unable to verify timestamp for Ark.Arktool.dll
*** ERROR: Module load completed but symbols could not be loaded for Ark.Arktool.dll
*** WARNING: Unable to verify timestamp for ACE.5.6.0.dll
*** ERROR: Module load completed but symbols could not be loaded for ACE.5.6.0.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for gfcppcache.dll -
*** WARNING: Unable to verify timestamp for msvcr80.dll
*** WARNING: Unable to verify timestamp for ExtServices.RFAAdapter.dll
*** ERROR: Module load completed but symbols could not be loaded for ExtServices.RFAAdapter.dll
*** WARNING: Unable to verify timestamp for RFA7_SessionLayer100.dll
*** ERROR: Module load completed but symbols could not be loaded for RFA7_SessionLayer100.dll
Unable to load image D:\eqtg\processes\Autofit\1.22.0-ST-c\RFA7_SSLED_Adapter100.dll, Win32 error 0n2
*** WARNING: Unable to verify timestamp for RFA7_SSLED_Adapter100.dll
*** ERROR: Module load completed but symbols could not be loaded for RFA7_SSLED_Adapter100.dll

FAULTING_IP:
+0
16bbd320 ?? ???

EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 16bbd320
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000008
Parameter[1]: 16bbd320
Attempt to execute non-executable address 16bbd320

DEFAULT_BUCKET_ID: WRONG_SYMBOLS

PROCESS_NAME: NET.Jedi.JediApp.exe

ERROR_CODE: (NTSTATUS) 0xc0000005 - 0x%08lx

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - 0x%08lx

EXCEPTION_PARAMETER1: 00000008

EXCEPTION_PARAMETER2: 16bbd320

WRITE_ADDRESS: 16bbd320

FOLLOWUP_IP:
customLL_GemfireRC+1cbd2
132dcbd2 ?? ???

FAILED_INSTRUCTION_ADDRESS:
+1cbd2
16bbd320 ?? ???

APPLICATION_VERIFIER_FLAGS: 0

APP: net.jedi.jediapp.exe

MANAGED_STACK: !dumpstack -EE
No export dumpstack found

PRIMARY_PROBLEM_CLASS: WRONG_SYMBOLS

BUGCHECK_STR: APPLICATION_FAULT_WRONG_SYMBOLS

LAST_CONTROL_TRANSFER: from 132dcbd2 to 16bbd320

STACK_TEXT:
WARNING: Frame IP not in any known module. Following frames may be wrong.
07934e94 132dcbd2 00000001 7c829dc9 16bea3a8 0x16bbd320
07934eb4 132c272b 16bea3a8 07934f24 00000000 customLL_GemfireRC+0x1cbd2
07934ec8 132c266a 167ab688 00000000 16beb938 customLL_GemfireRC+0x272b
07934f24 132c3702 167ab688 00014e78 07934e7c customLL_GemfireRC+0x266a
07934f6c 04bbd5e2 00000001 0793888c 16c1d988 customLL_GemfireRC+0x3702
07934f8c 04b84edb 16c1d988 00000000 00000000 customLL_Common+0x3d5e2
07934fa0 04b93a04 0793887c 07938880 00000103 customLL_Common+0x4edb
07934fb8 78aa2dc7 04bf0438 00000039 07934fd8 customLL_Common+0x13a04
0793894c 04baca6a 07938ba4 09a99f80 09a99f9c msvcr100!_NLG_Return
07938cb8 04bad410 07939134 07939f90 0793904c customLL_Common+0x2ca6a
07939148 04c6278d 07939f88 07939f90 09a990e8 customLL_Common+0x2d410
0793a050 04cd6749 0793acdc 000007da 00000012 UMI_CommonImplcustomLL+0x278d
0793acf0 04cd6b95 0793b7ac 000007da 00000012 UMI_CommonImpl+0x46749
0793b990 04ca25ed 0793dcd4 000007da 00000012 UMI_CommonImpl+0x46b95
0793ddf8 04ca3721 0793e0ec 05bc9c00 05bd6cc0 UMI_CommonImpl+0x125ed
0793de70 04cfca3a 0793e0ec 05bc9c00 05bd6cc0 UMI_CommonImpl+0x13721
0793e118 04cd1b1f 0793e350 16bd2c88 16c20d18 UMI_CommonImpl+0x6ca3a
0793e1f8 0d3a601d 0793e350 16bd2c88 16c20d18 UMI_CommonImpl+0x41b1f
0793e390 7916b956 00691f02 0c52376c 67b89896 0xd3a601d
0793e3cc 0d3a58bc 0793e41c 009a98a0 0e90bc3c clr!PreStubWorker+0x165
0793e43c 79141af9 791b5d2a 76fd28c3 00000000 0xd3a58bc
0793e440 791b5d2a 76fd28c3 00000000 0322dd8c clr!_EH_epilog3_catch_GS+0xa
0793e620 0d3a4ef5 0793e668 0793e670 0793e678 clr!MethodDesc::MakeJitWorker+0x5ca
0793e6d8 7916b956 00691f02 0656b128 00000000 0xd3a4ef5
0793e714 0d38f2c5 00000001 51eb851f 40a80d38 clr!PreStubWorker+0x165
0793e8a4 79141af9 7915f6a4 76fd2457 03090000 0xd38f2c5
0793e8a8 7915f6a4 76fd2457 03090000 17deb9ac clr!_EH_epilog3_catch_GS+0xa
0793e97c 00000000 17df07e0 00000000 00000000 clr!VirtualCallStubManager::ResolveWorker+0x5ea

SYMBOL_STACK_INDEX: 1

SYMBOL_NAME: customLL_gemfirerc+1cbd2

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: customLL_GemfireRC

IMAGE_NAME: customLL.GemfireRC.dll

DEBUG_FLR_IMAGE_TIMESTAMP: 52fc9b18

STACK_COMMAND: ~10s; .ecxr ; kb

FAILURE_BUCKET_ID: WRONG_SYMBOLS_c0000005_customLL.GemfireRC.dll!Unknown

BUCKET_ID: APPLICATION_FAULT_WRONG_SYMBOLS_BAD_IP_customLL_gemfirerc+1cbd2

Followup: MachineOwner

According to the stack trace, it seems like that the application crashed in the custom module CustomLL_GemfireRC

By adding the symbol file info of this custom module to the Symbol File Path, we may be able to see the exact place to cause this crash issue.