High CPU usage by AexSvc.exe, replication failure

Products

• IT Management Suite

Issue/Introduction

Replication in a hierarchy fails.

Two symptoms:

1. Excessive CPU utilization on child:
"9/13/2018 12:59:55 PM","[SYSTEM]
 [app cpu: 99%, ram: 724.01 MB / 4%, uptime: 27.18:46:41.0028452]
 [ns cpu: 100%, ram: 4.70 GB / 29%, uptime: 18.16:17:43.3085937]
 [sql cpu: 1%, ram: 11.86 GB / 74.1% (Available physical memory is high), cpu history %: 1 / 4 / 5 / 6 / 3 / 1 / 2 / 1 / 1]
 [ns machine: *********** (V), ram: 16.00 GB, cpu: 8x2400Mhz, versions: 8.1.4528.0, assembly: 8.1.4528.0, hierarchy: child]
 [sql machine: 10.117.80.76 (V), ram: 16.00 GB, cpu: 8x8, affinity: 2 (AUTO), version: 11.0.5058.0 / Enterprise Edition (64-bit) / SP2, trip: 0]
 [pc physical: 24879, virtual: 9330, managed: 331, connectivity: 333, hierarchy: 1, ps: 2, ts: 1]
 [.NET 4.0.30319.42000]","PerformanceSensor","AeXSVC.exe","236","Informational"

2. Failure to initiate job or update replication status:
"9/13/2018 3:01:33 PM","Unable to schedule replication to the specified server (Server: **********.***********.com, Exception: Altiris.NS.Exceptions.AeXException: The local credential username itms could not connect to the local server http://**********.***********.com/Altiris/NS/

-------------------------

Restart of the services (iiserest, Altiris Service restart) helps to resolve problem temporarily.

Memory dump of the process utilizing high CPU shows a large number of replication threads in this exact state:

0:058> !clrstack
OS Thread Id: 0x1c20 (58)
        Child SP               IP Call Site
0000007daf48e4d8 00007ff9d8920bba [GCFrame: 0000007daf48e4d8]
0000007daf48e5c0 00007ff9d8920bba [GCFrame: 0000007daf48e5c0]
0000007daf48e5f8 00007ff9d8920bba [HelperMethodFrame: 0000007daf48e5f8] System.Threading.Monitor.Enter(System.Object)
0000007daf48e6f0 00007ff9cdf4c42d System.Net.ConnectionGroup.FindConnection(System.Net.HttpWebRequest, System.String, Boolean ByRef) [f:\dd\NDP\fx\src\net\System\Net\_ConnectionGroup.cs @ 478]
0000007daf48e780 00007ff9cdf4b763 System.Net.ServicePoint.SubmitRequest(System.Net.HttpWebRequest, System.String) [f:\dd\NDP\fx\src\net\System\Net\ServicePoint.cs @ 399]
0000007daf48e7e0 00007ff9cdf4ac19 System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint) [f:\dd\NDP\fx\src\net\System\Net\HttpWebRequest.cs @ 4161]
0000007daf48e840 00007ff9cdf28a2b System.Net.HttpWebRequest.GetRequestStream(System.Net.TransportContext ByRef) [f:\dd\NDP\fx\src\net\System\Net\HttpWebRequest.cs @ 1559]
0000007daf48e8b0 00007ff9cdf28735 System.Net.HttpWebRequest.GetRequestStream() [f:\dd\NDP\fx\src\net\System\Net\HttpWebRequest.cs @ 1487]
0000007daf48e8e0 00007ff9cd3c1e37 System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(System.String, System.Object[])
0000007daf48e960 00007ff971a56855 Altiris.NS.Replication.dll!Unknown
0000007daf48e9a0 00007ff971a42eeb Altiris.NS.Replication.dll!Unknown
0000007daf48eb20 00007ff971a42b31 Altiris.NS.Replication.dll!Unknown
0000007daf48ec90 00007ff971a40f69 Altiris.NS.Replication.dll!Unknown
0000007daf48ed70 00007ff971a3ca0f Altiris.NS.Replication.dll!Unknown
0000007daf48ee90 00007ff971a3c78b Altiris.NS.Replication.dll!Unknown
0000007daf48eed0 00007ff9cf1131d3 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 954]
0000007daf48efa0 00007ff9cf113064 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 902]
0000007daf48efd0 00007ff9cf113032 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 891]
0000007daf48f020 00007ff9cfabef9c System.Threading.ThreadHelper.ThreadStart(System.Object) [f:\dd\ndp\clr\src\BCL\system\threading\thread.cs @ 93]
0000007daf48f278 00007ff9d02a6bc3 [GCFrame: 0000007daf48f278]
0000007daf48f5c8 00007ff9d02a6bc3 [DebuggerU2MCatchHandlerFrame: 0000007daf48f5c8]
 

"9/13/2018 3:01:33 PM","Unable to schedule replication to the specified server (Server: **********.***********.com, Exception: Altiris.NS.Exceptions.AeXException: The local credential username itms could not connect to the local server http://**********.***********..com/Altiris/NS/

Environment

ITMS 8.x

Cause

Known issue in .NET Framework in HttpWebRequest class.

Resolution

Workaround to the issue is to increase number of connections in web.config file. Does not resolve problem, but allows running server for longer without restart.

1. Open [C:\Program Files\Altiris\Notification Server\Web\Web.config]
2. Add this section under </configSections>  NOTE:  Put the following UNDER "/configSections" and not inside of it.
   
   <system.net>
       <connectionManagement>
         <add address = "*" maxconnection = "200" />
       </connectionManagement>
     </system.net>
    
3. Restart IIS.