APM 10.x, 2x
"[ERROR] [IntroscopeAgent.Agent] System.ComponentModel.Win32Exception (0x80004005): Access is denied"
at System.Diagnostics.ProcessManager.OpenThread(Int32 threadId, Int32 access)
at System.Diagnostics.ProcessThread.GetThreadTimes()
at System.Diagnostics.ProcessThread.get_TotalProcessorTime()
at com.wily.introscope.agent.feature.CpuTimingFeature.PutStartDataPoint(InvocationData data)
Suggestions:
1) Set introscope.agent.cpumethodtimebreakdown.enable=false in their introscopeagent.profile and restart IIS or the .NET App
2) Try commenting out all the corresponding SetTracerParameter lines in required.pbd and MVC.pbd that would set "cputiming true"? e.g.
in required.pbd:
#SetTracerParameter: PageInfoTracer cputiming true
in MVC.pbd:
#SetTracerParameter: DotNetMVCFrontendMarker cputiming true
#SetTracerParameter: DotNetMVCControllerAggregator cputiming true
#SetTracerParameter: ASPNETCoreAsyncFrontendTracer cputiming true
#SetTracerParameter: MVCControllerBlamePointTracer cputiming true
#SetTracerParameter: MVCControllerActionBlamePointTracer cputiming true
#SetTracerParameter: DotNetMVCFrontendMarkerAsync cputiming true
Back up the original IntroscopeAgent.profile, required.pbd and MVC.pbd before making these test configuration changes and restart the monitored .NET app
"Failed to CoCreate profiler” Profiler CLSID: '{D6E0BA92-3BC3-45ff-B9CC-B4B5AB7190BC}
the above message indicates that there is another .NET profiler already installed. Only one .NET profiler can run at a time, see previous point.
"System.InvalidProgramException: Common Language Runtime detected an invalid program" "…cannot be activated due to an exception during compilation".
Disable “WCFRuntimeTracing” in the webservices.pbd as below:
#TurnOn: WCFRuntimeTracing
Then restart IIS or the monitored standalone .NET application.
Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x5215df96
Faulting module name: KERNELBASE.dll, version: 6.3.9600.17415, time stamp: 0x54505737
Exception code: 0xe0434352
Fault offset: 0x0000000000008b9c
Faulting process id: 0x47e0
Faulting application start time: 0x01d138b4c21a8ef9
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 1f2322d9-a4a8-11e5-80f8-005056b772c9
1. Open the AGENT_HOME\toggles-typical.pbd.
2. Disable socket instrumentation as below.
#TurnOn: SocketTracing
3. Restart IIS
For example after installing .NET Agent version 10.5.2
If not listed, you need to register it manually, drag and drop AGENT_HOME\wily\bin\wily.Agent.dll into C:\Windows\assembly
Open Command Prompt as Administrator, run: set
The output should be similar to the one below:
Cor_Enable_Profiling=0x1
COR_PROFILER={5F048FC6-251C-4684-8CCA-76047B02AC98}
com.wily.introscope.agentprofile=<install_dir>\wily\IntroscopeAgent.profile
If you are instrumenting a .NET standalone app, make sure to run:
<AGENT_HOME>\wily\wilypermission.exe <AGENT_HOME>\wily <your application>
For example:
<AGENT_HOME>\wily\wilypermission.exe <AGENT_HOME>\wily mytestapp.exe
-Put activity to the .NET Application, then verify that the agent has been attached to the .NET process
- Run:
tasklist /m wily*
Here is the output you should see:
Image Name PID Modules
========================= ======== ============================================
PerfMonCollectorAgent.exe 1300 wily.NativeProfiler.dll, wily.Agent.dll
w3wp.exe 4000 wily.NativeProfiler.dll, wily.Agent.dll,
wily.WebServicesAgent.ext.dll
Open the IntroscopeAgent.profile
Verify the property:
agentManager.url.1=<your EM>:<EM port>
Uncomment the property:
The below logs should be created for each instrumented application:
-AutoProbe.*.log
-IntroscopeAgent.*.log
-nativeprofiler_*.log
This test will help you confirm if the issue is related to the application.
Open the IntroscopeAgent.profile, comment the "introscope.agent.dotnet.monitorApplications" property as below:
#introscope.agent.dotnet.monitorApplications=
Only 1 .NET profiler can run at a time , the Broadcom APM .NET Agent GUI is {5F048FC6-251C-4684-8CCA-76047B02AC98}
How to verify this condition:
- Open "Command Prompt" as Administrator,
- Run the command: REG QUERY HKLM /f "COR_PROFILER" /s
Or redirect the output to a file: REG QUERY HKLM /f "COR_PROFILER" /s > apm_netagent_regquery_corproofiler.txt
In the above example AVICODE, SCOM or Microsoft Monitoring Agent is preventing the .NET Agent to run as expected.
SCOM GUID = AD5651A8-B5C8-46ca-A11B-E82AEC2B8E78
To resolve this issue you need to uninstall the "Microsoft Monitoring Agent" from Add/Remove programs from the affected server(s).
See also:
APM .NET - Avicode conflicts with the .NET agent
https://knowledge.broadcom.com/external/article/115892
Make sure the reflection setting is added to all calls to the Kernel init.
For example:
public NinjectDependencyResolver()
{
kernel = new StandardKernel(new {UseReflectionBasedInjection = true;});
AddBindings();
}
..
For more information see:
http://stackoverflow.com/questions/5772860/exception-when-using-ninject
http://stackoverflow.com/questions/11989300/running-ninject-3-on-medium-trust-level
http://stackoverflow.com/questions/19770350/invalidprogramexception-profiling-a-webforms-app-using-ninject
For exact details how to enable UseReflectionBasedInjection, contact Ninject support or communities: http://www.ninject.org/community.html
a) From the Workstation or Webview, check if the Agent metric clamp has been reached, expand the branch:
Custom Metric Host (virtual)
- Custom Metric Process (virtual)
- Custom Metric Agent (virtual)([email protected])(SuperDomain)
- Agents
- Host
- Process
- AgentName
looks at the values for : “is Clamped” metric, it should return zero(0)
b) To monitor this condition for the Perform Collector, you need to enable verbose logging:
- Open the logging.config.xml, enable verblose logging as below:
<level value="VERBOSE"/>
- Restart IIS or the .NET standalone process .
- if the permon metric clamp has been reached you will see in the the Agent log the below message:
[VERBOSE] [IntroscopeAgent.PerfMonService] Metric limit of xx has been reached
- Uncomment the below property and increase the value as needed, then restart the "CA APM PerfMon Collector Service".
#introscope.agent.perfmon.metric.limit=1000
You can try to isolate the issue by adjusting the Agent configuration as below:
TEST#1: Find out if the issue is related to the Agent instrumentation
-Open the IntroscopeAgent.profile , set introscope.autoprobe.enable=false
-Stop the Windows Perfmon collector service.
-Restart IIS or the .NET instrumented application.
If the problem occurs, contact Broadcom Support
If the problem doesn’t’ persists, proceed with TEST#2
TEST#2: Disable SOAP/HTTP header injection
Header insertion is enabled by default, however, some applications can be impacted by this change.
In this case you can disable header insertion as below:
a) enable back instrumentation : set introscope.autoprobe.enable=true
b) Open webservices.pbd , disable WebServicesCorrelationTracing group as below:
#TurnOn: WebServicesCorrelationTracing
c) Open MCV.pbd , disable GenericClientCorrelationTracing group as below:
#TurnOn: GenericClientCorrelationTracing
Save the files
d) Restart IIS or the .NET instrumented application.
If the problem doesn’t’ persists, proceed with TEST#3
TEST#3: Disable the entire Webservices and MVC instrumentation
Open webservices.pbd and MVC.pbd and disable all the lines that start with TurnOn as below:
#TurnOn: WebServicesProducerTracing
#TurnOn: WebServicesClientTracing
#TurnOn: WCFServerFaultTracing
#TurnOn: WCFClientFaultTracing
.....
Save the files
Restart IIS or the .NET instrumented application.
If the problem occurs, you need need to enable back the above traces groups 1 by 1 until you identify the root cause
If the problem doesn’t’ persists, proceed with TEST#4
TEST#4: Limit the # of monitored applications, try to monitor only specific application by updating the "introscope.agent.dotnet.monitorApplications" property
For example, to limit the agent to monitor IIS and the below DummyWinApp.exe application, you would need to update the monitorApplications property as below:
introscope.agent.dotnet.monitorApplications=w3wp.exe,aspnet_wp.exe,DummyWinApp.exe
TEST#5: Reduce Perform metric collection in IntroscopeAgent.profile to prevent CPU overhead / spikes
1.Stop the PerfMonCollectorAgent service
If the problem doesn’t’ persists, enable back PerfMonCollectorAgent service and try to reduce perfmon data collection as below
2.Open the IntroscopeAgent.profile, set:
introscope.agent.perfmon.category.browseEnabled=false
Purpose: to confirm whether disabling perfmon category browsing would significantly reduce the CPU overhead
introscope.agent.perfmon.metric.pollIntervalInSeconds=150
Purpose: to confirm if the CPU spikes is related to the polling interval and # of metrics, after increasing the value from 15 to 150 check if the CPU spikes occurs around every 150 seconds
introscope.agent.perfmon.metric.filterPattern=|Processor|*|*,Process|{osprocessname}|*,|ASP.NET*|*
Purpose: you might need to reduce the # of perfmon metrics being gathered. Depending on your application load the default setting could instruct the agent to collect a huge amount of metrics, introscope.agent.perfmon.metric.filterPattern=|Processor|*|*,|.NET Data Provider*|*|*,|.NET CLR*|{osprocessname}|*,|.NET CLR Data|*|*,|Process|{osprocessname}|*,|ASP.NET*|*
3.Start the PerfMonCollectorAgent service and verify if the issue persists.
TEST#6: Try to disable the Agent Traces feature:
com.wily.introscope.agent.blame.transaction.doTransactionTrace=false
This is hot property
Enable DEBUG logging:
- Open the logging.config.xml
- Change the logging level from INFO to DEBUG:
<level value="DEBUG"/>
- Restart IIS or NET app
NOTE: if you cannot restart IIS, open the IntroscopeAgent.profile, make a change and save the file, this will force the update in the logging.config.xml to be taken into account
If IIS or NET app is crashing:
- open the IntroscopeAgent.profile
- set introscope.nativeprofiler.logBytecode=true and introscope.nativeprofiler.logAllMethodsNoticed=true
Try to reproduce the issue and collect the below information
If IIS or NET app is crashing, collect dump: there are multiple ways to capture dumps on.NET process.
Option 1: Bring up Task Manager, find the .NET process with the memory issue, and then right click on the process to select Create Dump File option in its context menu.
Option 2: (RECOMMENDED) Create a dump file when a process crashes as documented here : https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dumps
Essentially it just requires the setting of some system variables via Control Panel -> System -> Advanced System Setting -> Environment Variables
COMPlus_DbgEnableMiniDump value 1
COMPlus_DbgMiniDumpType value 1
COMPlus_DbgMiniDumpName value C:\TT\coredump (set this to a location of your choice
- ensure sufficient free space the files
can be very large)
COMPlus_CreateDumpDiagnostics value 1
Then when a .Net process crashes it will write the dump file to the location defined above.
As this can create a lot of data I would suggest that once you have captured a dump from the failure that you then disable the dump creation by setting the variable COMPlus_DbgEnableMiniDump back to 0
The above all assumes of course that there is a process crash - which is what you indicated - when the failure occurs.
Ideally we recommend performing the above in a test environment assuming that the problem is reproducible there.