Troubleshooting stopped RiskFabric processing jobs

Products

Information Centric Analytics Data Loss Prevention Core Package

Issue/Introduction

You notice while using Information Centric Analytics (ICA) that data are stale or missing on one or more dashboards. Data in the Analyzer may also be stale or missing. When you check the job status of the RiskFabric Processing or RiskFabric Intraday Processing jobs in the console by navigating to Admin > Integration > Job Status, the Last Run Outcome for the job is Cancelled. Using SQL Server Management Studio (SSMS) to check the job status under SQL Server Agent > Jobs, you see an error similar to the following logged in the job history:

Executed as user: <domain>\<account-name> The step was canceled (stopped) as the result of a stop job request.

NOTE: a stopped or cancelled processing job differs in nature from a processing failure. If the processing job completed with failures or appears to be hung, this article is not applicable.

Environment

Version : 6.x

Component : Microsoft SQL Server

Cause

A stopped job can be caused for a number of reasons, including:

The SQL Server host was manually shutdown or restarted while the RiskFabric Processing or RiskFabric Intraday Processing job was running
The SQL Server host underwent a power loss or other physical issue that caused a shutdown while the RiskFabric Processing or RiskFabric Intraday Processing job was running
NOTE: An abrupt power loss will not log a job cancellation message. Instead, after restarting, the SQL Server Agent log will show the job is still in a running state despite the underlying process having been terminated by the shutdown. A graceful shutdown command by an uninterruptible power supply (UPS) caused by a power loss will log the cancellation in the same manner as a shutdown command issued by an administrator.
The SQL Server or SQL Server Agent service was manually stopped while the RiskFabric Processing or RiskFabric Intraday Processing job was running
The SQL Server or SQL Server Agent service crashed while the RiskFabric Processing or RiskFabric Intraday Processing job was running due to an out-of-memory condition or other application fault

Resolution

The following resources can assist with determining the cause of a job cancellation:

SQL Server logs
- In SSMS, navigate to Management > SQL Server Logs
- In Windows Explorer or a Command Prompt on the SQL Server host, navigate to:
```
...\Program Files\Microsoft SQL Server\MSSQL<n>.<instance-name>\Log
```
Windows Logs
- Application
  - Using Event Viewer, navigate to Event Viewer (Local) > Windows Logs > Application and look for events related to either SQLAgent$<instance-name> or MSSQL$<instance-name>
  - Look for events under either of the Task Category values Server or Service Control
  - The following example shows the progression of events caused by a manual shutdown of the SQL Server and SQL Server Agent services:
```
SQLServerAgent service successfully stopped.
SQL Server is terminating in response to a 'stop' request from Service Control Manager. This is an informational message only. No user action is required.
Service Broker manager has shut down.
.NET Framework runtime has been stopped.
SQL Trace was stopped due to server shutdown. Trace ID = '1'. This is an informational message only; no user action is required.
```
- System
  - Using Event Viewer, navigate to Event Viewer (Local) > Windows Logs > System and look for events from the source Service Control Manager
  - The following examples shows events caused by a manual shutdown of the SQL Server and SQL Server Agent services:
```
The SQL Server Agent (<instance-name>) service entered the stopped state.
The SQL Server (<instance-name>) service entered the stopped state.
```

As noted in the Cause section of this article, server failures due to power loss will not generate log entries in the SQL Server and SQL Server Agent logs; however, Windows will log recovery events from an unexpected shutdown and this can be used to determine whether the job was cancelled due to a power failure. To review these logs, follow this procedure:

Open Event Viewer
Navigate to Event Viewer (Local) > Windows Logs > System
Right-click System and select Filter Current Log...
The Filter Current Log window will open
To view all user-initiated shutdown and restart events:
1. From the Event sources dropdown menu, check the option User32
2. In the box labelled <All Event IDs>, enter the value 1074
3. Click the OK button to apply the filter
4. As you review each event, note the Shutdown Type listed under the General tab
To view all unexpected shutdowns:
1. In the box labelled <All Event IDs>, enter the value 6008
2. Click the OK button to apply the filter

In addition to reviewing both Windows and SQL Server logging, contact the server administrator and/or database administrator (DBA) for assistance with determining why the job was stopped. The server may have been shutdown recently for maintenance, software installation, or testing. Services may have been stopped for the same reasons. These events may be evident in the logs but lack context or an explanation that your server and database administrators can provide.