Exit Code 10 on Metric Collector processing
search cancel

Exit Code 10 on Metric Collector processing

book

Article ID: 257365

calendar_today

Updated On:

Products

Information Centric Analytics

Issue/Introduction

The SQL Server Agent logs a failure for the nightly RiskFabric Processing job step Metric Collector. The log message will differ depending upon the underlying cause of the failure, but in all cases returns exit code 10.

No Error Message

The message logged does not indicate the cause of failure. The body of the SQL Server Agent log message shows multiple successful metric scrapes and is similar to the following:

2022-09-12 14:34:40,279 [1:INFO] Connections.GetAnalysisServicesConnectionString() Init: Retrieving Default Analysis Server from Risk Fabric Portal Settings...
2022-09-12 14:34:40,279 [1:INFO] Connections.GetAnalysisServicesConnectionString() Init: SELECT [Name],[Value] FROM dbo.[PortalSettings] WHERE Name in ('ASDBServer','ASDBDatabase')
2022-09-12 14:34:41,592 [1:INFO] Program.Main() Init: **SUCCESS: Database validated.'
2022-09-12 14:34:41,592 [1:INFO] Connections.CreateAnalysisServicesDatabaseWorker() Analysis Server Connection String: Data Source=<SSAS_hostname>;Catalog=RiskFabric
2022-09-12 14:34:43,561 [1:INFO] Connections.CreateAnalysisServicesDatabaseWorker() Init: Default Analysis Server is used when ServerName/Database is not overriden in Metrics table.
2022-09-12 14:34:43,561 [1:INFO] Program.DoRunSnapshotCollector() Metric: Loading all Metrics into cache.
2022-09-12 14:34:48,811 [1:INFO] MdxQueryProcessor.QueryMdxKpiByOrganizationAndCountry() Metric: Snapshot for Analyzer Metric named 'Multiple Methods Per Hour - Users Last 30 Days' scraped successfully.
2022-09-12 14:34:51,663 [1:INFO] MdxQueryProcessor.QueryMdxKpiByOrganizationAndCountry() Metric: Snapshot for Analyzer Metric named 'Same File Name - Blocked then Allowed - Users Last 30 Days' scraped successfully.
2022-09-12 14:34:54,523 [1:INFO] MdxQueryProcessor.QueryMdxKpiByOrganizationAndCountry() Metric: Snapshot for Analyzer Metric named 'Same Match Count - Blocked then Allowed - Users Last 30 Days' scraped successfully.
2022-09-12 14:34:57,521 [1:INFO] MdxQueryProcessor.QueryMdxKpiByOrganizationAndCountry() Metric: Snapshot for Analyzer Metric named 'Large Files to USB - Users Last 30 Days' scraped successfully.
2022-09-12 14:35:00,426 [1:INFO] MdxQueryProcessor.QueryMdxKpiByOrganizationAndCountry() Metric: Snapshot for Analyzer Metric named 'High Match Count Unusual - Users Last 30 Days' scraped successfully.
2022-09-12 14:35:03,331 [1:INFO] MdxQueryProcessor.QueryMdxKpiByOrganizationAndCountry() Metric: Snapshot for Analyzer Metric named '5 Email Blocks / Day  - Users Last 30 Days' scraped successfully.
2022-09-12 14:35:06,331 [1:INFO] MdxQueryProcessor.QueryMdxKpiByOrganizationAndCountry() Metric: Snapshot for Analyzer Metric named 'High Severity Unusual - Users Last 30 Days' scraped successfully.
2022-09-12 14:35:09,308 [1:INFO] MdxQueryProcessor.QueryMdxKpiByOrganizationAndCountry() Metric: Snapshot for Analyzer Metric named 'Unusual Policy Volume - Users Last 30 Days' scraped successfully.
2022-09-12 14:35:12,421 [1:INFO] MdxQueryProcessor.QueryMdxKpiByOrganizationAndCountry() Metric: Snapshot for Analyzer Metric named 'Repeat Policy Offender - Users Last 30 Days' scraped successfully.
2022-09-12 14:35:15,328 [1:INFO] MdxQueryProcessor.QueryMdxKpiByOrganizationAndCountry() Metric: Snapshot for Analyzer Metric named 'Team Broken Process - Users Last 30 Days' scraped successfully.
2022-09-12 14:35:18,218 [1:INFO] MdxQueryProcessor.QueryMdxKpiByOrganizationAndCountry() Metric: Snapshot for Analyzer Metric named 'Department Broken Process - Users Last 30 Days' scraped successfully.
2022-09-12 14:35:21,245 [1:INFO] MdxQueryProcessor.QueryMdxKpiByOrganizationAndCountry() Metric: Snapshot for Analyzer Metric named 'Contractor Unusual - Users Last 30 Days' scraped successfully.
2022-09-12 14:35:24,122 [1:INFO] MdxQueryProcessor.QueryMdxKpiByOrganizationAndCountry() Metric: Snapshot for Analyzer Metric named 'Planned Departure - Users Last 30 Days' scraped successfully.
2022-09-12 14:35:26,982 [1:INFO] MdxQueryProcessor.QueryMdxKpiByOrganizationAndCountry() Metric: Snapshot for Analyzer Metric named 'Terminated Employee - Users Last 30 Days' scraped successfully.
2022-09-12 14:35:30,767 [1:INFO] MdxQueryProcessor.QueryMdxKpiByOrganizationAndCountry() Metric: Snapshot for Analyzer Metric named 'Total DIM Incidents Last 30 Days' scraped successfully.
2022-09-12 14:35:35,414...  Process Exit Code 10.  The step failed.

The failure occurs after a 10-minute wait (600000 ms), as recorded in the Metric Collector log file (RiskFabricCollector.<yyyyMMdd>.log):

2022-10-27 02:25:10,555 [1:INFO] MetricSnapshotController.CopySnapshotsToMetricHistory() Metric: bulk copy completed successfully.
2022-10-27 02:35:10,627 [1:ERROR] ...

Enabling the SQL Server Agent setting Include execution trace messages reveals the following:

2022-10-27 02:35:10,627 [1:ERROR] Program.Main() Exception System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.

Error Message

An error message similar to the following is logged in the SQL Server Agent history log:

2025-04-08 18:46:23,200 [1:INFO] Connections.createConnectionSettings() Init: Retrieving Default Analysis Server from Risk Fabric Portal Settings...
2025-04-08 18:46:23,200 [1:INFO] Connections.createConnectionSettings() Init: SELECT [Name],[Value] FROM dbo.[PortalSettings] WHERE Name in ('ASDBServer','ASDBDatabase','AnalysisServicesTimeout','SQLCommandTimeOut')
2025-04-08 18:46:23,998 [1:INFO] Program.Main() Init: **SUCCESS: Database validated.'
2025-04-08 18:46:23,998 [1:INFO] Connections.CreateAnalysisServicesDatabaseWorker() Analysis Server Connection String: Data Source=<SSAS_hostname>;Catalog=RiskFabric
2025-04-08 18:46:28,123 [1:ERROR] Connections.CreateAnalysisServicesDatabaseWorker() ASDB Connection Exception: {0}  Microsoft.AnalysisServices.AdomdClient.AdomdConnectionException: A connection cannot be made. Ensure that the server is running. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it <SSAS_hostname_or_IP>     at System.Net.Sockets.TcpClient..ctor(String hostname, Int32 port)     at Microsoft.AnalysisServices.AdomdClient.XmlaClient.GetTcpClient(ConnectionInfo connectionInfo)     --- End of inner exception stack trace ---     at Microsoft.AnalysisServices.AdomdClient.XmlaClient.GetTcpClient(ConnectionInfo connectionInfo)     at Microsoft.AnalysisServices.AdomdClient.XmlaClient.OpenTcpConnection(ConnectionInfo connectionInfo)     at Microsoft.AnalysisServices.AdomdClient.XmlaClient.OpenConnection(ConnectionInfo connectionInfo, Boolean& isSessionTokenNeeded)     at Microsoft.AnalysisServices.AdomdClient.XmlaClient.Connect(ConnectionInfo connectionInfo, Boolean beginSession)     at Microsoft.AnalysisServices.AdomdClient.AdomdConnection.XmlaClientProvider.Connect(Boolean toIXMLA)     at Microsoft.AnalysisServices.AdomdClient.AdomdConnection.XmlaClientProvider.Microsoft.AnalysisServices.AdomdClient.AdomdConnection.IXmlaClientProviderEx.ConnectXmla()     at Microsoft.AnalysisServices.AdomdClient.AdomdConnection.ConnectToXMLA(Boolean createSession, Boolean isHTTP)     at Microsoft.AnalysisServices.AdomdClient.AdomdConnection.Open()     at RiskFabric.Collector.ASDatabase..ctor(String asdbConnectionString, Int32 commandTimeout)     at RiskFabric.Collector.DataAccess.Connections.CreateAnalysisServicesDatabaseWorker(ConnectionSettings connectionSettings)
2025-04-08 18:46:28,123 [1:INFO] Connections.CreateAnalysisServicesDatabaseWorker() Continue
2025-04-08 18:46:28,123 [1:ERROR] Program.Main() Exception  System.NullReferenceException: Object reference not set to an instance of an object.     at RiskFabric.Collector.DataAccess.MetricSnapshotController.GetQueryProcessorFactory(DateTime snapshotSeed)     at RiskFabricCollector.Program.DoRunSnapshotCollector()     at RiskFabricCollector.Program.Main(String[] args).  Process Exit Code 10.  The step failed.

A similar message is logged in the Metric Collector log (RiskFabricCollector.<yyyyMMdd>.log):

2025-03-05 00:52:58,774 [1:INFO] Connections.createConnectionSettings() Init: Retrieving Default Analysis Server from Risk Fabric Portal Settings...
2025-03-05 00:52:58,774 [1:INFO] Connections.createConnectionSettings() Init: SELECT [Name],[Value] FROM dbo.[PortalSettings] WHERE Name in ('ASDBServer','ASDBDatabase','AnalysisServicesTimeout','SQLCommandTimeOut')
2025-03-05 00:52:59,039 [1:INFO] Program.Main() Init: **SUCCESS: Database validated.'
2025-03-05 00:52:59,055 [1:INFO] Connections.CreateAnalysisServicesDatabaseWorker() Analysis Server Connection String: Data Source=<SSAS_hostname>;Catalog=RiskFabric
2025-03-05 00:53:00,607 [1:INFO] Connections.CreateAnalysisServicesDatabaseWorker() Init: Default Analysis Server is used when ServerName/Database is not overriden in Metrics table.
2025-03-05 00:53:00,607 [1:INFO] Program.DoRunSnapshotCollector() Metric: Loading all Metrics into cache.
2025-03-05 00:53:03,018 [1:ERROR] MdxQueryProcessor.GetASDatabase() Metric [Multiple Methods Per Hour - Users Last 30 Days]: ASDB Connection Exception: A connection cannot be made. Ensure that the server is running.

Environment

Version : 6.x

Component : Metric Collector

Cause

If no error message is logged with exit code 10, the cause is a hardcoded timeout set in the RiskFabric Collector executable. SQL metric commands inherit a 10 minute timeout, which overrides the SQL connection timeout value. This inherited value was meant to be 0, or no timeout. This timeout may be reached while processing metrics for an unusually large number of days, as might occur following a system recovery or a delay in processing spanning multiple days.

If an error message is logged with exit code 10, connection failures can be caused by any of the following conditions:

  1. The SQL Server Analysis Services (SSAS) service is not running
  2. A firewall or network rule is blocking connections to the SSAS instance or server
  3. The SSAS instance in which the RiskFabric OLAP cube is deployed does not match the hostname, instance name, or port number defined in the Risk Fabric general portal setting Analysis Services Server

Resolution

Beginning with 6.6 MP1 (6.6.1.0), the timeouts in the Metric Collector process will use the general porting settings SQL Command Timeout (Seconds) and Analysis Services Timeout in Seconds, which are user-configurable.

For connectivity issues:

  • Ensure the SSAS service is running on the server hosting the RiskFabric OLAP cube.
  • Verify firewall rules on the SSAS server allow connections to the SSAS service. The default SSAS instance listens on port 2383. Named instances use dynamic ports. For more information, refer to Microsoft document Configure the Windows Firewall to Allow Analysis Services Access
  • Verify the Analysis Services Server portal setting is configured with the correct hostname, instance name, and port. You can do this by logging into the Risk Fabric console and navigating to Admin > Settings > General > Advanced Settings, or by executing the following statement in SQL Server Management Studio (SSMS): 
    SELECT DisplayName, [Value] FROM RiskFabric.dbo.PortalSettings WHERE [Name] IN (N'ASDBServer', N'ASDBDatabase');

Additional Information

The Metric Collector log file (RiskFabricCollector.<yyyyMMdd>.log) is located on the server hosting ICA's database utilities. By default, this should be the server hosting the RiskFabric relational database. The path is:

%ProgramData%\BayDynamics\Logs
Powered by Wolken Software