Client Task Agent is unable to register consistently if bandwidth throttling is enable on Task Server
search cancel

Client Task Agent is unable to register consistently if bandwidth throttling is enable on Task Server

book

Article ID: 239889

calendar_today

Updated On:

Products

IT Management Suite

Issue/Introduction

A customer has reported that one of his Task Servers have some problems in getting task registration to itself or Client Task Agent on client machines can't register to it in a consistent matter.

After waiting for a while, Task Server will allow/succeed registration.

We can see that the Task Server is trying to register to itself but then it aborts the process.

Entry 1:

Operation 'Direct: Get' failed. 
Protocol: HTTP 
Host: <TS01>.example.net:80 
Path: /Altiris/TaskManagement/ClientTask/GetClientData.aspx 
Connection Id: 23.2656 
Communication profile Id: {8060C289-22B3-45A5-9082-B705234B94CE} 
Throttling: 2 30 0 
Error type: Network error 
Error code: An established connection was aborted by the software in your host machine (10053) 
Error note: SocketIOStrategySyncSelect::Receive error
-----------------------------------------------------------------------------------------------------
Date: 4/4/2022 8:05:09 AM, Tick Count: 1446718 (00:24:06.7180000), Size: 671 B
Process: AtrsHost.exe (2656), Thread ID: 5112, Module: AeXNetComms.dll
Priority: 1, Source: NetworkOperation

Entry 2:

One or more errors occurred.
   [System.AggregateException @ mscorlib]
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Altiris.DotNetLib.Interop.AeXNetComms.AeXAsyncRequest.Execute()
   at Altiris.DotNetLib.Interop.AeXNetComms.Web.AeXWebRequest.Execute(String requestUriString, Guid cnnProfileId, TimeSpan timeout, Boolean isPost, String data)
   at Altiris.DotNetLib.Helpers.AtrsHttpOps.Execute(String url, Guid cnnProfileId, TimeSpan timeout, Boolean isPost, String data)
   at Altiris.ClientTask.Server.Communication.NotificationServerWebConnection.PostToNotificationServerWithRetry(String url, Guid cnnProfileId, String data, Int32 nMaxAttempts, TimeSpan timeout)

The request has been aborted by timeout.
   [Altiris.DotNetLib.Interop.AeXNetComms.Exceptions.AeXRequestAbortedByTimeoutException @ mscorlib]
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Altiris.DotNetLib.Interop.AeXNetComms.AeXAsyncRequest.<ExecuteCallback>d__56.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Altiris.DotNetLib.Interop.AeXNetComms.AeXTransportInstance.<ExecuteRequestCallback>d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Altiris.DotNetLib.Interop.AeXNetComms.AeXPooledTransportInstance.<ExecuteRequestCallback>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Altiris.DotNetLib.Interop.AeXNetComms.AeXTransportManager.<ExecuteRequest>d__44.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Altiris.DotNetLib.Interop.AeXNetComms.AeXAsyncRequest.<ExecuteAsync>d__55.MoveNext()

Exception logged from: 
   at Altiris.DotNetLib.Logging.AtrsLog.ExceptionMessage(String message, Exception exception)
   at Altiris.ClientTask.Server.Logging.NSAgentLog.ReportMessage(Severity severity, String moduleName, String source, Exception exception, String message, Object[] arguments)
   at Altiris.ClientTask.Server.Communication.NotificationServerWebConnection.PostToNotificationServerWithRetry(String url, Guid cnnProfileId, String data, Int32 nMaxAttempts, TimeSpan timeout)
   at Altiris.ClientTask.Server.Communication.NotificationServerWebConnection.PostToNotificationServer(String url, Guid cnnProfileId, String postData, String& respData, Boolean& dataEncryped, Int32 nMaxAttempts, TimeSpan timeout)
   at Altiris.ClientTask.Server.Communication.NotificationServerWebConnection.Altiris.TaskManagement.Common.ClientTask.Communication.IInternalNotificationServerConnection.GetClientData(Guid resourceGuid)
   at Altiris.ClientTask.Server.ClientTaskServer.RegisterClient(AltirisResourceGuid resourceGuid, IPAddress ipAddress, Boolean bLastResort, Boolean bBeingActive, String configurationXml, String crc, Hashtable& output, IAuthAgentConnection wsConnection, Version version, Nullable`1 resTypeGuid, String systemType)
   at Altiris.ClientTask.Server.ClientTaskServer.ProcessRemoteRegisterClient(Hashtable input, Hashtable& output)
   at Altiris.DotNetLib.RemotingRequestService.Altiris.DotNetLib.IRemotingRequestService.HandleRequest(String service, Hashtable parameters)
   at SyncInvokeHandleRequest(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
   at System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext currentOperationContext)
   at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)
   at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)
   at System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(IAsyncResult result)
   at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
   at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
   at System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceiveAsyncResult.OnReceive(IAsyncResult result)
   at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
   at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
   at System.ServiceModel.Channels.SynchronizedMessageSource.ReceiveAsyncResult.OnReceiveComplete(Object state)
   at System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(Object state)
   at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
   at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
   at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
   at System.Net.Security.NegotiateStream.ProcessFrameBody(Int32 readBytes, Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.NegotiateStream.ReadCallback(AsyncProtocolRequest asyncRequest)
   at System.Net.AsyncProtocolRequest.CompleteRequest(Int32 result)
   at System.Net.FixedSizeReader.CheckCompletionBeforeNextRead(Int32 bytes)
   at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
   at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
   at System.ServiceModel.Channels.ConnectionStream.IOAsyncResult.OnAsyncIOComplete(Object state)
   at System.ServiceModel.Channels.PipeConnection.OnAsyncReadComplete(Boolean haveResult, Int32 error, Int32 numBytes)
   at System.ServiceModel.Channels.OverlappedContext.CompleteCallback(UInt32 error, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
   at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)


-----------------------------------------------------------------------------------------------------
Date: 4/4/2022 8:05:09 AM, Tick Count: 1446718 (00:24:06.7180000), Size: 7.19 KB
Process: AtrsHost.exe (2656), Thread ID: 5112, Module: AtrsHost.exe
Priority: 1, Source: NotificationServerWebConnection

 

On the Task Server, you may see entries like these as well referencing the IP Address of the machine that is trying to register to it:

Entry 1:

Failed to handle web service request: RegisterClient
This request operation sent to net.pipe://localhost/rss/ClientTaskServerRemotingRequestService did not receive a reply within the configured timeout (00:00:26.4936426).  The time allotted to this operation may have been a portion of a longer timeout.  This may be because the service is still processing the operation or because the service was unable to send a reply message.  Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client.
   [System.TimeoutException @ mscorlib]

Server stack trace: 
   at System.ServiceModel.Dispatcher.DuplexChannelBinder.SyncDuplexRequest.WaitForReply(TimeSpan timeout)
   at System.ServiceModel.Dispatcher.DuplexChannelBinder.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Altiris.DotNetLib.IRemotingRequestService.HandleRequest(String service, Hashtable parameters)
   at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.CallTaskServer(String method, Hashtable input)

Exception logged from: 
   at Altiris.DotNetLib.Logging.AtrsLog.ExceptionMessage(String message, Exception exception)
   at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.Loggers.<>c.<.cctor>b__7_1(String message, Exception ex, Int32 errorCode, Int32 severity)
   at Altiris.DotNetLib.Constructs.ThrottledLogAction`2.Execute(T1 arg1, T2 arg2, Int32 errorCode)
   at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.CallTaskServer(String method, Hashtable input)
   at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.ExecuteService(String serviceName, Hashtable input, IAuthAgentConnection wsConnection)
   at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.ProcessRegister(XmlTextWriter wr, NameValueCollection queryValues, Stream requestStream, Int32 contentLength, String address, IAuthAgentConnection wsConnection)
   at Altiris.ClientTask.Server.HttpHandlers.Register.WriteResponse(XmlTextWriter wr)
   at Altiris.DotNetLib.Threading.StringBuilderCache.ToXml(Action`1 fu)
   at Altiris.TaskManagement.Common.XmlHttp.BaseXmlXmlHttpCallback.WriteResponseRaw(XmlTextWriter xwr)
   at Altiris.TaskManagement.Common.XmlHttp.BaseXmlHttpCallback.ProcessRequest(HttpContext context)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
   at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)
   at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)
   at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
   at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)


-----------------------------------------------------------------------------------------------------
Date: 4/4/2022 8:04:16 AM, Tick Count: 1393968 (00:23:13.9680000), Size: 4.57 KB
Process: w3wp.exe (1060), Thread ID: 1460, Module: w3wp.exe
Priority: 2, Source: WebClientApiManager

Entry 2:

Unable to process request from: ###.###.###.###/POST/8.6.3269.0 (Failed to handle web service request: RegisterClient, 0x8000FFFF, FailedServerGeneralFailure)
-----------------------------------------------------------------------------------------------------
Date: 4/4/2022 8:04:16 AM, Tick Count: 1393968 (00:23:13.9680000), Size: 407 B
Process: w3wp.exe (1060), Thread ID: 1460, Module: w3wp.exe
Priority: 2, Source: Altiris.ClientTask.Server.HttpHandlers.Register

Environment

ITMS 8.5, 8.6

Cause

Known issue. Bandwidth throttling was enable on the Task Server and set to a value like 30%. The agent logs should show what value is in use:

Operation 'Direct: Get' failed. 
Protocol: HTTP 
Host: <TS01>.example.net:80 
Path: /Altiris/TaskManagement/ClientTask/GetClientData.aspx 
Connection Id: 23.2656 
Communication profile Id: {8060C289-22B3-45A5-9082-B705234B94CE} 
Throttling: 2 30 0

Bandwidth throttling has an effect on Task Server (TS) to Notification Server (NS) calls. It significantly decreases TS->NS communication performance which causes "registration calls abortion" due to timeouts on the Task Server.

Resolution

This issue has been reported to the Broadcom Development team. A fix has been added to our next release: ITMS 8.6 RU3.

Starting with ITMS 8.6 RU3, Bandwidth Throttling for TS calls is disabled in our code.

 

Workaround:

Disable bandwidth throttling on the affected Task Server or don't use a too small value for it.

 

Powered by Wolken Software