ALERT: Some images may not load properly within the Knowledge Base Article. If you see a broken image, please right-click and select 'Open image in a new tab'. We apologize for this inconvenience.

Client Task Agent is unable to register consistently if bandwidth throttling is enable on Task Server

book

Article ID: 239889

calendar_today

Updated On:

Products

IT Management Suite

Issue/Introduction

A customer has reported that one of his Task Servers have some problems in getting task registration to itself or Client Task Agent on client machines can't register to it in a consistent matter.

After waiting for a while, Task Server will allow/succeed registration.

We can see that the Task Server is trying to register to itself but then it aborts the process.

Entry 1:

Operation 'Direct: Get' failed. 
Protocol: HTTP 
Host: TS01.domain.net:80 
Path: /Altiris/TaskManagement/ClientTask/GetClientData.aspx 
Connection Id: 23.2656 
Communication profile Id: {8060C289-22B3-45A5-9082-B705234B94CE} 
Throttling: 2 30 0 
Error type: Network error 
Error code: An established connection was aborted by the software in your host machine (10053) 
Error note: SocketIOStrategySyncSelect::Receive error
-----------------------------------------------------------------------------------------------------
Date: 4/4/2022 8:05:09 AM, Tick Count: 1446718 (00:24:06.7180000), Size: 671 B
Process: AtrsHost.exe (2656), Thread ID: 5112, Module: AeXNetComms.dll
Priority: 1, Source: NetworkOperation

Entry 2:

One or more errors occurred.
   [System.AggregateException @ mscorlib]
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Altiris.DotNetLib.Interop.AeXNetComms.AeXAsyncRequest.Execute()
   at Altiris.DotNetLib.Interop.AeXNetComms.Web.AeXWebRequest.Execute(String requestUriString, Guid cnnProfileId, TimeSpan timeout, Boolean isPost, String data)
   at Altiris.DotNetLib.Helpers.AtrsHttpOps.Execute(String url, Guid cnnProfileId, TimeSpan timeout, Boolean isPost, String data)
   at Altiris.ClientTask.Server.Communication.NotificationServerWebConnection.PostToNotificationServerWithRetry(String url, Guid cnnProfileId, String data, Int32 nMaxAttempts, TimeSpan timeout)

The request has been aborted by timeout.
   [Altiris.DotNetLib.Interop.AeXNetComms.Exceptions.AeXRequestAbortedByTimeoutException @ mscorlib]
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Altiris.DotNetLib.Interop.AeXNetComms.AeXAsyncRequest.<ExecuteCallback>d__56.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Altiris.DotNetLib.Interop.AeXNetComms.AeXTransportInstance.<ExecuteRequestCallback>d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Altiris.DotNetLib.Interop.AeXNetComms.AeXPooledTransportInstance.<ExecuteRequestCallback>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Altiris.DotNetLib.Interop.AeXNetComms.AeXTransportManager.<ExecuteRequest>d__44.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Altiris.DotNetLib.Interop.AeXNetComms.AeXAsyncRequest.<ExecuteAsync>d__55.MoveNext()

Exception logged from: 
   at Altiris.DotNetLib.Logging.AtrsLog.ExceptionMessage(String message, Exception exception)
   at Altiris.ClientTask.Server.Logging.NSAgentLog.ReportMessage(Severity severity, String moduleName, String source, Exception exception, String message, Object[] arguments)
   at Altiris.ClientTask.Server.Communication.NotificationServerWebConnection.PostToNotificationServerWithRetry(String url, Guid cnnProfileId, String data, Int32 nMaxAttempts, TimeSpan timeout)
   at Altiris.ClientTask.Server.Communication.NotificationServerWebConnection.PostToNotificationServer(String url, Guid cnnProfileId, String postData, String& respData, Boolean& dataEncryped, Int32 nMaxAttempts, TimeSpan timeout)
   at Altiris.ClientTask.Server.Communication.NotificationServerWebConnection.Altiris.TaskManagement.Common.ClientTask.Communication.IInternalNotificationServerConnection.GetClientData(Guid resourceGuid)
   at Altiris.ClientTask.Server.ClientTaskServer.RegisterClient(AltirisResourceGuid resourceGuid, IPAddress ipAddress, Boolean bLastResort, Boolean bBeingActive, String configurationXml, String crc, Hashtable& output, IAuthAgentConnection wsConnection, Version version, Nullable`1 resTypeGuid, String systemType)
   at Altiris.ClientTask.Server.ClientTaskServer.ProcessRemoteRegisterClient(Hashtable input, Hashtable& output)
   at Altiris.DotNetLib.RemotingRequestService.Altiris.DotNetLib.IRemotingRequestService.HandleRequest(String service, Hashtable parameters)
   at SyncInvokeHandleRequest(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
   at System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext currentOperationContext)
   at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)
   at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)
   at System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(IAsyncResult result)
   at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
   at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
   at System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceiveAsyncResult.OnReceive(IAsyncResult result)
   at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
   at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
   at System.ServiceModel.Channels.SynchronizedMessageSource.ReceiveAsyncResult.OnReceiveComplete(Object state)
   at System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(Object state)
   at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
   at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
   at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
   at System.Net.Security.NegotiateStream.ProcessFrameBody(Int32 readBytes, Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.NegotiateStream.ReadCallback(AsyncProtocolRequest asyncRequest)
   at System.Net.AsyncProtocolRequest.CompleteRequest(Int32 result)
   at System.Net.FixedSizeReader.CheckCompletionBeforeNextRead(Int32 bytes)
   at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
   at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
   at System.ServiceModel.Channels.ConnectionStream.IOAsyncResult.OnAsyncIOComplete(Object state)
   at System.ServiceModel.Channels.PipeConnection.OnAsyncReadComplete(Boolean haveResult, Int32 error, Int32 numBytes)
   at System.ServiceModel.Channels.OverlappedContext.CompleteCallback(UInt32 error, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
   at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)


-----------------------------------------------------------------------------------------------------
Date: 4/4/2022 8:05:09 AM, Tick Count: 1446718 (00:24:06.7180000), Size: 7.19 KB
Process: AtrsHost.exe (2656), Thread ID: 5112, Module: AtrsHost.exe
Priority: 1, Source: NotificationServerWebConnection

 

On the Task Server, you may see entries like these as well referencing the IP Address of the machine that is trying to register to it:

Entry 1:

Failed to handle web service request: RegisterClient
This request operation sent to net.pipe://localhost/rss/ClientTaskServerRemotingRequestService did not receive a reply within the configured timeout (00:00:26.4936426).  The time allotted to this operation may have been a portion of a longer timeout.  This may be because the service is still processing the operation or because the service was unable to send a reply message.  Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client.
   [System.TimeoutException @ mscorlib]

Server stack trace: 
   at System.ServiceModel.Dispatcher.DuplexChannelBinder.SyncDuplexRequest.WaitForReply(TimeSpan timeout)
   at System.ServiceModel.Dispatcher.DuplexChannelBinder.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Altiris.DotNetLib.IRemotingRequestService.HandleRequest(String service, Hashtable parameters)
   at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.CallTaskServer(String method, Hashtable input)

Exception logged from: 
   at Altiris.DotNetLib.Logging.AtrsLog.ExceptionMessage(String message, Exception exception)
   at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.Loggers.<>c.<.cctor>b__7_1(String message, Exception ex, Int32 errorCode, Int32 severity)
   at Altiris.DotNetLib.Constructs.ThrottledLogAction`2.Execute(T1 arg1, T2 arg2, Int32 errorCode)
   at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.CallTaskServer(String method, Hashtable input)
   at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.ExecuteService(String serviceName, Hashtable input, IAuthAgentConnection wsConnection)
   at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.ProcessRegister(XmlTextWriter wr, NameValueCollection queryValues, Stream requestStream, Int32 contentLength, String address, IAuthAgentConnection wsConnection)
   at Altiris.ClientTask.Server.HttpHandlers.Register.WriteResponse(XmlTextWriter wr)
   at Altiris.DotNetLib.Threading.StringBuilderCache.ToXml(Action`1 fu)
   at Altiris.TaskManagement.Common.XmlHttp.BaseXmlXmlHttpCallback.WriteResponseRaw(XmlTextWriter xwr)
   at Altiris.TaskManagement.Common.XmlHttp.BaseXmlHttpCallback.ProcessRequest(HttpContext context)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
   at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)
   at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)
   at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
   at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)


-----------------------------------------------------------------------------------------------------
Date: 4/4/2022 8:04:16 AM, Tick Count: 1393968 (00:23:13.9680000), Size: 4.57 KB
Process: w3wp.exe (1060), Thread ID: 1460, Module: w3wp.exe
Priority: 2, Source: WebClientApiManager

Entry 2:

Unable to process request from: 10.89.234.159/POST/8.6.3269.0 (Failed to handle web service request: RegisterClient, 0x8000FFFF, FailedServerGeneralFailure)
-----------------------------------------------------------------------------------------------------
Date: 4/4/2022 8:04:16 AM, Tick Count: 1393968 (00:23:13.9680000), Size: 407 B
Process: w3wp.exe (1060), Thread ID: 1460, Module: w3wp.exe
Priority: 2, Source: Altiris.ClientTask.Server.HttpHandlers.Register

Cause

Known issue. Bandwidth throttling was enable on the Task Server and set to a value like 30%. The agent logs should show what value is in use:

Operation 'Direct: Get' failed. 
Protocol: HTTP 
Host: TS01.domain.net:80 
Path: /Altiris/TaskManagement/ClientTask/GetClientData.aspx 
Connection Id: 23.2656 
Communication profile Id: {8060C289-22B3-45A5-9082-B705234B94CE} 
Throttling: 2 30 0

Bandwidth throttling has an effect on Task Server (TS) to Notification Server (NS) calls. It significantly decreases TS->NS communication performance which causes "registration calls abortion" due to timeouts on the Task Server.

Environment

ITMS 8.5, 8.6

Resolution

This issue has been reported to the Broadcom Development team. A fix has been added to our next release: ITMS 8.6 RU3.

Starting with ITMS 8.6 RU3, Bandwidth Throttling for TS calls is disabled in our code.

 

Workaround:

Disable bandwidth throttling on the affected Task Server or don't use a too small value for it.

 

Attachments