High CPU usage by AexSvc.exe, replication failure

Products

• IT Management Suite

Issue/Introduction

Replication in a hierarchy fails.

Two symptoms:

1. Excessive CPU utilization on child:
"9/13/2018 12:59:55 PM","[SYSTEM]
 [app cpu: 99%, ram: 724.01 MB / 4%, uptime: 27.18:46:41.0028452]
 [ns cpu: 100%, ram: 4.70 GB / 29%, uptime: 18.16:17:43.3085937]
 [sql cpu: 1%, ram: 11.86 GB / 74.1% (Available physical memory is high), cpu history %: 1 / 4 / 5 / 6 / 3 / 1 / 2 / 1 / 1]
 [ns machine: *********** (V), ram: 16.00 GB, cpu: 8x2400Mhz, versions: 8.1.4528.0, assembly: 8.1.4528.0, hierarchy: child]
 [sql machine: 10.117.80.76 (V), ram: 16.00 GB, cpu: 8x8, affinity: 2 (AUTO), version: 11.0.5058.0 / Enterprise Edition (64-bit) / SP2, trip: 0]
 [pc physical: 24879, virtual: 9330, managed: 331, connectivity: 333, hierarchy: 1, ps: 2, ts: 1]
 [.NET 4.0.30319.42000]","PerformanceSensor","AeXSVC.exe","236","Informational"

2. Failure to initiate job or update replication status:
"9/13/2018 3:01:33 PM","Unable to schedule replication to the specified server (Server: **********.***********.com, Exception: Altiris.NS.Exceptions.AeXException: The local credential username itms could not connect to the local server http://**********.***********.com/Altiris/NS/

-------------------------

Restart of the services (iiserest, Altiris Service restart) helps to resolve problem temporarily.

Memory dump of the process utilizing high CPU shows a large number of replication threads in this exact state:

 

0:058> !clrstack
OS Thread Id: 0x1c20 (58)
        Child SP               IP Call Site
0000007daf48e4d8 00007ff9d8920bba [GCFrame: 0000007daf48e4d8]
0000007daf48e5c0 00007ff9d8920bba [GCFrame: 0000007daf48e5c0]
0000007daf48e5f8 00007ff9d8920bba [HelperMethodFrame: 0000007daf48e5f8] System.Threading.Monitor.Enter(System.Object)
0000007daf48e6f0 00007ff9cdf4c42d System.Net.ConnectionGroup.FindConnection(System.Net.HttpWebRequest, System.String, Boolean ByRef) [f:\dd\NDP\fx\src\net\System\Net\_ConnectionGroup.cs @ 478]
0000007daf48e780 00007ff9cdf4b763 System.Net.ServicePoint.SubmitRequest(System.Net.HttpWebRequest, System.String) [f:\dd\NDP\fx\src\net\System\Net\ServicePoint.cs @ 399]
0000007daf48e7e0 00007ff9cdf4ac19 System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint) [f:\dd\NDP\fx\src\net\System\Net\HttpWebRequest.cs @ 4161]
0000007daf48e840 00007ff9cdf28a2b System.Net.HttpWebRequest.GetRequestStream(System.Net.TransportContext ByRef) [f:\dd\NDP\fx\src\net\System\Net\HttpWebRequest.cs @ 1559]
0000007daf48e8b0 00007ff9cdf28735 System.Net.HttpWebRequest.GetRequestStream() [f:\dd\NDP\fx\src\net\System\Net\HttpWebRequest.cs @ 1487]
0000007daf48e8e0 00007ff9cd3c1e37 System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(System.String, System.Object[])
0000007daf48e960 00007ff971a56855 Altiris.NS.Replication.dll!Unknown
0000007daf48e9a0 00007ff971a42eeb Altiris.NS.Replication.dll!Unknown
0000007daf48eb20 00007ff971a42b31 Altiris.NS.Replication.dll!Unknown
0000007daf48ec90 00007ff971a40f69 Altiris.NS.Replication.dll!Unknown
0000007daf48ed70 00007ff971a3ca0f Altiris.NS.Replication.dll!Unknown
0000007daf48ee90 00007ff971a3c78b Altiris.NS.Replication.dll!Unknown
0000007daf48eed0 00007ff9cf1131d3 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 954]
0000007daf48efa0 00007ff9cf113064 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 902]
0000007daf48efd0 00007ff9cf113032 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 891]
0000007daf48f020 00007ff9cfabef9c System.Threading.ThreadHelper.ThreadStart(System.Object) [f:\dd\ndp\clr\src\BCL\system\threading\thread.cs @ 93]
0000007daf48f278 00007ff9d02a6bc3 [GCFrame: 0000007daf48f278]
0000007daf48f5c8 00007ff9d02a6bc3 [DebuggerU2MCatchHandlerFrame: 0000007daf48f5c8]
 

"9/13/2018 3:01:33 PM","Unable to schedule replication to the specified server (Server: **********.***********.com, Exception: Altiris.NS.Exceptions.AeXException: The local credential username itms could not connect to the local server http://**********.***********..com/Altiris/NS/

Environment

ITMS 8.x

Cause

Known issue in .NET Framework in HttpWebRequest class.

Possibly related article from MS: https://support.microsoft.com/en-us/help/2497453/fix-application-may-stop-responding-after-several-http-requests-are-ab

Resolution

Workaround to the issue is to increase number of connections in web.config file. Does not resolve problem, but allows running server for longer without restart.
 

1. Open [C:\Program Files\Altiris\Notification Server\Web\Web.config]
2. Add this section under </configSections>  NOTE:  Put the following UNDER "/configSections" and not inside of it.
   
   <system.net>
       <connectionManagement>
         <add address = "*" maxconnection = "200" />
       </connectionManagement>
     </system.net>
    
3. Restart IIS.