.Net Agent crashing applications
Article ID: 250695
Updated On:
Products
Issue/Introduction
We have multiple scenarios where the .NET agent is crashing our application. We are seeing multiple errors in the application logs:
First chance exception - 0XE0434352 caused by thread with System ID: 16152. Type = com.wily.util.adt.DequeueTimedOutException. DetailID = 1
Below is what our observation is so far:
1. When the Application pool is configured to run in Integrated mode, The Application crashes, Users get a 500 and on the second or third try they can get working again.
2. When the Application pool is configured to be classic, The Application crashes and never recovers, at this point someone has to go on the server to recycle/stop/start the app pool.
These are the logs we see in the evening viewer for each setup. I have also attached application logs with the ticket.
When app pool in Classic mode
You will see two entries in the event viewer
1st Error
Log Name: Application
Source: ASP.NET 4.0.30319.0
Date: 2022-04-20 2:47:26 PM
Event ID: 1325
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: myserverA.mycompany.com
Description:
Failed to initialize the AppDomain:/LM/W3SVC/1/ROOT/vital/UtilityMenu/SweepUtil
Exception: System.InvalidOperationException
Message: The configuration system has already been initialized.
StackTrace: at System.Configuration.ConfigurationManager.SetConfigurationSystem(IInternalConfigSystem configSystem, Boolean initComplete)
at System.Configuration.Internal.InternalConfigSettingsFactory.System.Configuration.Internal.IInternalConfigSettingsFactory.SetConfigurationSystem(IInternalConfigSystem configSystem, Boolean initComplete)
at System.Web.Configuration.HttpConfigurationSystem.EnsureInit(IConfigMapPath configMapPath, Boolean listenToFileChanges, Boolean initComplete)
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException)
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel)
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel)
at System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironment(String appId, IApplicationHost appHost, HostingEnvironmentParameters hostingParameters)
at System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironmentAndReportErrors(String appId, IApplicationHost appHost, HostingEnvironmentParameters hostingParameters)
Event Xml:
<Event xmlns=http://schemas.microsoft.com/win/2004/08/events/event>
<System>
<Provider Name="ASP.NET 4.0.30319.0" />
<EventID Qualifiers="49152">1325</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2022-04-20T18:47:26.000000000Z" />
<EventRecordID>3686310</EventRecordID>
<Channel>Application</Channel>
<Computer>myserverA.mycompany.com</Computer>
<Security />
</System>
<EventData>
<Data>Failed to initialize the AppDomain:/LM/W3SVC/1/ROOT/vital/UtilityMenu/SweepUtil
Exception: System.InvalidOperationException
Message: The configuration system has already been initialized.
StackTrace: at System.Configuration.ConfigurationManager.SetConfigurationSystem(IInternalConfigSystem configSystem, Boolean initComplete)
at System.Configuration.Internal.InternalConfigSettingsFactory.System.Configuration.Internal.IInternalConfigSettingsFactory.SetConfigurationSystem(IInternalConfigSystem configSystem, Boolean initComplete)
at System.Web.Configuration.HttpConfigurationSystem.EnsureInit(IConfigMapPath configMapPath, Boolean listenToFileChanges, Boolean initComplete)
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException)
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel)
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel)
at System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironment(String appId, IApplicationHost appHost, HostingEnvironmentParameters hostingParameters)
at System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironmentAndReportErrors(String appId, IApplicationHost appHost, HostingEnvironmentParameters hostingParameters)</Data>
</EventData>
</Event>
2nd Error
Log Name: Application
Source: ASP.NET 4.0.30319.0
Date: 2022-04-20 2:47:26 PM
Event ID: 1088
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: myserverA.mycompany.com
Description:
Failed to execute request because the App-Domain could not be created. Error: 0x80131509
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="ASP.NET 4.0.30319.0" />
<EventID Qualifiers="49152">1088</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2022-04-20T18:47:26.000000000Z" />
<EventRecordID>3686311</EventRecordID>
<Channel>Application</Channel>
<Computer>myserverA.mycompany.com</Computer>
<Security />
</System>
<EventData>
<Data>0x80131509 </Data>
</EventData>
</Event>
When the App Pool is set to Integrated mode
You see three errors in the event viewer
1st error
Log Name: Application
Source: ASP.NET 4.0.30319.0
Date: 2022-04-20 11:22:16 AM
Event ID: 1325
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: myserverA.mycompany.com
Description:
Failed to initialize the AppDomain:/LM/W3SVC/1/ROOT/LandingPage/XU
Exception: System.InvalidOperationException
Message: The configuration system has already been initialized.
StackTrace: at System.Configuration.ConfigurationManager.SetConfigurationSystem(IInternalConfigSystem configSystem, Boolean initComplete)
at System.Configuration.Internal.InternalConfigSettingsFactory.System.Configuration.Internal.IInternalConfigSettingsFactory.SetConfigurationSystem(IInternalConfigSystem configSystem, Boolean initComplete)
at System.Web.Configuration.HttpConfigurationSystem.EnsureInit(IConfigMapPath configMapPath, Boolean listenToFileChanges, Boolean initComplete)
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException)
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel)
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel)
at System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironment(String appId, IApplicationHost appHost, HostingEnvironmentParameters hostingParameters)
at System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironmentAndReportErrors(String appId, IApplicationHost appHost, HostingEnvironmentParameters hostingParameters)
Event Xml:
<Event xmlns=http://schemas.microsoft.com/win/2004/08/events/event>
<System>
<Provider Name="ASP.NET 4.0.30319.0" />
<EventID Qualifiers="49152">1325</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2022-04-20T15:22:16.000000000Z" />
<EventRecordID>3685837</EventRecordID>
<Channel>Application</Channel>
<Computer>myserverA.mycompany.com</Computer>
<Security />
</System>
<EventData>
<Data>Failed to initialize the AppDomain:/LM/W3SVC/1/ROOT/LandingPage/XU
Exception: System.InvalidOperationException
Message: The configuration system has already been initialized.
StackTrace: at System.Configuration.ConfigurationManager.SetConfigurationSystem(IInternalConfigSystem configSystem, Boolean initComplete)
at System.Configuration.Internal.InternalConfigSettingsFactory.System.Configuration.Internal.IInternalConfigSettingsFactory.SetConfigurationSystem(IInternalConfigSystem configSystem, Boolean initComplete)
at System.Web.Configuration.HttpConfigurationSystem.EnsureInit(IConfigMapPath configMapPath, Boolean listenToFileChanges, Boolean initComplete)
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException)
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel)
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel)
at System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironment(String appId, IApplicationHost appHost, HostingEnvironmentParameters hostingParameters)
at System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironmentAndReportErrors(String appId, IApplicationHost appHost, HostingEnvironmentParameters hostingParameters)</Data>
</EventData>
</Event>
2nd error
Log Name: Application
Source: ASP.NET 4.0.30319.0
Date: 2022-04-20 11:22:16 AM
Event ID: 1325
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: myserverA.mycompany.com
Description:
An error occurred while trying to start an integrated application instance.
Exception: System.InvalidOperationException
Message: The configuration system has already been initialized.
StackTrace: at System.Configuration.ConfigurationManager.SetConfigurationSystem(IInternalConfigSystem configSystem, Boolean initComplete)
at System.Configuration.Internal.InternalConfigSettingsFactory.System.Configuration.Internal.IInternalConfigSettingsFactory.SetConfigurationSystem(IInternalConfigSystem configSystem, Boolean initComplete)
at System.Web.Configuration.HttpConfigurationSystem.EnsureInit(IConfigMapPath configMapPath, Boolean listenToFileChanges, Boolean initComplete)
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException)
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel)
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel)
at System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironment(String appId, IApplicationHost appHost, HostingEnvironmentParameters hostingParameters)
at System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironmentAndReportErrors(String appId, IApplicationHost appHost, HostingEnvironmentParameters hostingParameters)
at System.Web.Hosting.ApplicationManager.GetAppDomainWithHostingEnvironment(String appId, IApplicationHost appHost, HostingEnvironmentParameters hostingParameters)
at System.Web.Hosting.ApplicationManager.CreateObjectInternal(String appId, Type type, IApplicationHost appHost, Boolean failIfExists, HostingEnvironmentParameters hostingParameters)
at System.Web.Hosting.ProcessHost.StartApplication(String appId, String appPath, Object& runtimeInterface)
Event Xml:
<Event xmlns=http://schemas.microsoft.com/win/2004/08/events/event>
<System>
<Provider Name="ASP.NET 4.0.30319.0" />
<EventID Qualifiers="49152">1325</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2022-04-20T15:22:16.000000000Z" />
<EventRecordID>3685838</EventRecordID>
<Channel>Application</Channel>
<Computer>myserverA.mycompany.com</Computer>
<Security />
</System>
<EventData>
<Data>An error occurred while trying to start an integrated application instance.
Exception: System.InvalidOperationException
Message: The configuration system has already been initialized.
StackTrace: at System.Configuration.ConfigurationManager.SetConfigurationSystem(IInternalConfigSystem configSystem, Boolean initComplete)
at System.Configuration.Internal.InternalConfigSettingsFactory.System.Configuration.Internal.IInternalConfigSettingsFactory.SetConfigurationSystem(IInternalConfigSystem configSystem, Boolean initComplete)
at System.Web.Configuration.HttpConfigurationSystem.EnsureInit(IConfigMapPath configMapPath, Boolean listenToFileChanges, Boolean initComplete)
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException)
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel)
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel)
at System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironment(String appId, IApplicationHost appHost, HostingEnvironmentParameters hostingParameters)
at System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironmentAndReportErrors(String appId, IApplicationHost appHost, HostingEnvironmentParameters hostingParameters)
at System.Web.Hosting.ApplicationManager.GetAppDomainWithHostingEnvironment(String appId, IApplicationHost appHost, HostingEnvironmentParameters hostingParameters)
at System.Web.Hosting.ApplicationManager.CreateObjectInternal(String appId, Type type, IApplicationHost appHost, Boolean failIfExists, HostingEnvironmentParameters hostingParameters)
at System.Web.Hosting.ProcessHost.StartApplication(String appId, String appPath, Object& runtimeInterface)</Data>
</EventData>
</Event>
3rd error
Log Name: Application
Source: Microsoft-Windows-IIS-W3SVC-WP
Date: 2022-04-20 11:22:16 AM
Event ID: 2299
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: myserverA.mycompany.com
Description:
An application has reported as being unhealthy. The worker process will now request a recycle. Reason given: ASP.NET application initialization failed.. The data is the error.
Event Xml:
<Event xmlns=http://schemas.microsoft.com/win/2004/08/events/event>
<System>
<Provider Name="Microsoft-Windows-IIS-W3SVC-WP" Guid="{670080D9-742A-4187-8D16-41143D1290BD}" EventSourceName="W3SVC-WP" />
<EventID Qualifiers="49152">2299</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>0</Task>
<Opcode>0</Opcode>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2022-04-20T15:22:16.000000000Z" />
<EventRecordID>3685839</EventRecordID>
<Correlation />
<Execution ProcessID="0" ThreadID="0" />
<Channel>Application</Channel>
<Computer>myserverA.mycompany.com</Computer>
<Security />
</System>
<EventData>
<Data Name="Reason">ASP.NET application initialization failed.</Data>
<Binary>09151380</Binary>
</EventData>
</Event>
Environment
Release : SAAS
Component : Integration with APM
Resolution
Found that the property below was set to blank causing native Windows applications to be instrumented.
introscope.agent.dotnet.monitorApplications=w3wp.exe,aspnet_wp.exe
Since after disabling autoprobe it was found that everything works as expected. So confirmed that instrumentation was causing the crash.
Found that disabling additional correlation-related tracing flags below in MVC.pbd worked.
#TurnOn: AsyncHTTPBackendsTracing
#TurnOn: GenericClientCorrelationTracing
By disabling those tracers, what is lost in terms of metrics?
- Correlation will inject an extra header into the http or soap message request and your application may have a more strict security check, that is if they see some injection they software may not like it.
- If we disable the tracers, those headers will not appear and any violation will not be caught by their security check, but then you will lose the correlation of those transaction calls. The metrics will still come through, but the correlation will not be there, so when troubleshooting, it will be noticed in terms of the call sequence.
Feedback
