Error Spamming logs on Failed to call method: RegisterClient, AtrsHost service is unreachable

book

Article ID: 171731

calendar_today

Updated On:

Products

Management Platform (Formerly known as Notification Server) Task Server

Issue/Introduction

The customer started noticing the following NS log entries affecting Task Server:

Message 1:

Failed to call task server method: RegisterClient
The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.
[System.ServiceModel.CommunicationObjectFaultedException @ mscorlib]

Message 2:

The handler 'Register' is failed to process request. The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.
[System.ServiceModel.CommunicationObjectFaultedException @ mscorlib]

Message 3:

Failed to call method: RegisterClient, AtrsHost service is unreachable.
There was no endpoint listening at net.pipe://localhost/rss/ClientTaskServer that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.
[System.ServiceModel.EndpointNotFoundException @ mscorlib]

Failed to call task server method: RegisterClient
The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.
[System.ServiceModel.CommunicationObjectFaultedException @ mscorlib]

Server stack trace:
at System.ServiceModel.Channels.CommunicationObject.ThrowIfDisposedOrNotOpen()
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Altiris.DotNetLib.IRemotingRequestService.HandleRequest(String service, Hashtable parameters)
at Altiris.DotNetLib.RemotingRequestService.XmlCallClient.SendRequest(String service, Hashtable parameters)
at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.CallTaskServer(String method, Hashtable input)

Exception logged from:
at Altiris.Diagnostics.Logging.EventLog.ReportException(Int32 severity, String strMessage, String category, Exception exception, String footer)
at Altiris.NS.Logging.EventLog.ReportException(Int32 severity, String strMessage, String category, Exception exception)
at Altiris.TaskManagement.Logging.AltirisServerLog.ReportException(String moduleName, String source, Exception handledException, String message, Object[] arguments)
at Altiris.DotNetLib.Logging.AtrsLog.ReportExceptionWithSource(String source, Exception handledException, String message, Object[] arguments)
at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.CallTaskServer(String method, Hashtable input)
at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.ProcessRegister(XmlTextWriter wr, NameValueCollection queryValues, Stream requestStream, Int32 contentLength)
at Altiris.ClientTask.Server.HttpHandlers.Register.WriteResponse(XmlTextWriter wr)
at Altiris.DotNetLib.Threading.StringBuilderCache.ToXml(Action`1 fu)
at Altiris.TaskManagement.Common.XmlHttp.BaseXmlXmlHttpCallback.WriteResponseRaw(XmlTextWriter xwr)
at Altiris.TaskManagement.Common.XmlHttp.BaseXmlHttpCallback.ProcessRequest(HttpContext context)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)
at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)
at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)

**CEDUrlStart** :http://entced.symantec.com/entt?product=SMP&version=7.5.3153.0&language=en&module=IFrNby/8zZUzkf8jfR5gSyjv5K0QP3oR6lVaOtvMVTB1xqc6Ah6+31vUn7d+5C4CvfbIEET2vBq1v6IaCVzgLIoMEgznkjt5aD39R+nyFo8=&error=-128674299&build=**CEDUrlEnd**

-----------------------------------------------------------------------------------------------------
Date: 5/17/2018 12:43:27 PM, Tick Count: 102981812 (1.04:36:21.8120000), Host Name: OPRMGR05, Size: 4.14 KB
Process: w3wp (7968), Thread ID: 27, Module: w3wp.exe
Priority: 1, Source: Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.CallTaskServer

 


The handler 'Register' is failed to process request. The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.
[System.ServiceModel.CommunicationObjectFaultedException @ mscorlib]

Server stack trace:
at System.ServiceModel.Channels.CommunicationObject.ThrowIfDisposedOrNotOpen()
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Altiris.DotNetLib.IRemotingRequestService.HandleRequest(String service, Hashtable parameters)
at Altiris.DotNetLib.RemotingRequestService.XmlCallClient.SendRequest(String service, Hashtable parameters)
at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.CallTaskServer(String method, Hashtable input)

Exception logged from:
at Altiris.DotNetLib.Logging.AtrsLog.ExceptionMessage(String message, Exception exception)
at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.CallTaskServer(String method, Hashtable input)
at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.ProcessRegister(XmlTextWriter wr, NameValueCollection queryValues, Stream requestStream, Int32 contentLength)
at Altiris.ClientTask.Server.HttpHandlers.Register.WriteResponse(XmlTextWriter wr)
at Altiris.DotNetLib.Threading.StringBuilderCache.ToXml(Action`1 fu)
at Altiris.TaskManagement.Common.XmlHttp.BaseXmlXmlHttpCallback.WriteResponseRaw(XmlTextWriter xwr)
at Altiris.TaskManagement.Common.XmlHttp.BaseXmlHttpCallback.ProcessRequest(HttpContext context)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)
at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)
at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)

at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.CallTaskServer(String method, Hashtable input)
at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.ProcessRegister(XmlTextWriter wr, NameValueCollection queryValues, Stream requestStream, Int32 contentLength)
at Altiris.ClientTask.Server.HttpHandlers.Register.WriteResponse(XmlTextWriter wr)
at Altiris.DotNetLib.Threading.StringBuilderCache.ToXml(Action`1 fu)
at Altiris.TaskManagement.Common.XmlHttp.BaseXmlXmlHttpCallback.WriteResponseRaw(XmlTextWriter xwr)
at Altiris.TaskManagement.Common.XmlHttp.BaseXmlHttpCallback.ProcessRequest(HttpContext context)
-----------------------------------------------------------------------------------------------------
Date: 5/17/2018 12:43:27 PM, Tick Count: 102981812 (1.04:36:21.8120000), Host Name: OPRMGR05, Size: 4.06 KB
Process: w3wp (7968), Thread ID: 27, Module: w3wp.exe
Priority: 2, Source: Altiris.TaskManagement.Common.XmlHttp.BaseXmlHttpCallback.LogResponseException

 

Failed to call method: RegisterClient, AtrsHost service is unreachable.
There was no endpoint listening at net.pipe://localhost/rss/ClientTaskServer that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.
[System.ServiceModel.EndpointNotFoundException @ mscorlib]

Server stack trace:
at System.ServiceModel.Channels.PipeConnectionInitiator.GetPipeName(Uri uri)
at System.ServiceModel.Channels.NamedPipeConnectionPoolRegistry.NamedPipeConnectionPool.GetPoolKey(EndpointAddress address, Uri via)
at System.ServiceModel.Channels.CommunicationPool`2.TakeConnection(EndpointAddress address, Uri via, TimeSpan timeout, TKey& key)
at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Altiris.DotNetLib.IRemotingRequestService.HandleRequest(String service, Hashtable parameters)
at Altiris.DotNetLib.RemotingRequestService.XmlCallClient.SendRequest(String service, Hashtable parameters)
at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.CallTaskServer(String method, Hashtable input)

The pipe endpoint 'net.pipe://localhost/rss/ClientTaskServer' could not be found on your local machine.
[System.IO.PipeException @ ]


Exception logged from:
at Altiris.DotNetLib.Logging.AtrsLog.ExceptionMessage(String message, Exception exception)
at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.CallTaskServer(String method, Hashtable input)
at Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.ProcessRegister(XmlTextWriter wr, NameValueCollection queryValues, Stream requestStream, Int32 contentLength)
at Altiris.ClientTask.Server.HttpHandlers.Register.WriteResponse(XmlTextWriter wr)
at Altiris.DotNetLib.Threading.StringBuilderCache.ToXml(Action`1 fu)
at Altiris.TaskManagement.Common.XmlHttp.BaseXmlXmlHttpCallback.WriteResponseRaw(XmlTextWriter xwr)
at Altiris.TaskManagement.Common.XmlHttp.BaseXmlHttpCallback.ProcessRequest(HttpContext context)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)
at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)
at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)


-----------------------------------------------------------------------------------------------------
Date: 5/17/2018 12:43:27 PM, Tick Count: 102981953 (1.04:36:21.9530000), Host Name: OPRMGR05, Size: 4.52 KB
Process: w3wp (7968), Thread ID: 28, Module: w3wp.exe
Priority: 2, Source: Altiris.ClientTask.Server.HttpHandlers.WebClientApiManager.CallTaskServer

 

Cause

The overall situation is very likely because of startup code of TS, which did the update of the server status in DB and due to bug in some procedures, it failed - as a result, no remoting interfaces were created/open to allow connections from IIS.

Environment

ITMS 7.5 SP1 HF5

Resolution

This issue has been fixed on our ITMS 7.6 HF7, 8.0 HF2 and 8.1

There is a fix for those still in ITMS 7.5 SP1:

  1. Update "spResourceUpdateSummaryFullUpdateBulk" and "spResourceUpdateSummaryPartialUpdateBulk" with the attached ones:
    spResourceUpdateSummaryFullUpdateBulk_-_7_5.txt
    spResourceUpdateSummaryPartialUpdateBulk_-_7_5.txt


    Note: these stored procedure provided only work on 7.5 SP1 HF5. For a copy of the same ones for 7.6 HF7, please refer to TECH234772
  2. Restart AtrsHost service, Altiris Service and IIS.

Attachments

spResourceUpdateSummaryPartialUpdateBulk_-_7_5.txt get_app
spResourceUpdateSummaryFullUpdateBulk_-_7_5.txt get_app