data_engine intermittently raising "Bulk load data was expected but not sent. The batch will be terminated." alarms

book

Article ID: 131039

calendar_today

Updated On:

Products

DX Infrastructure Management NIMSOFT PROBES CA Unified Communications Monitor (NetQoS / UCM) CA Unified Infrastructure Management for z Systems

Issue/Introduction

data_engine 9.02HF3 is intermittently raising the following alarms which is cleared 20 to 30 seconds later: 

"Bulk load data was expected but not sent. The batch will be terminated."

"Insert bulk failed due to a schema change of the target table."

Cause

These alarms are generated by the ODBC/Native client driver while the data_engine probe attempts creating a rowset

Environment

UIM 9.02 (and earlier)
data_engine:  any version

MSSQL Server 2016 database

Resolution

The data_engine may generate an alarm that says "Insert bulk failed due to a schema change of the target table." This is most likely to happen during the housekeeping procedures which, among other things, are reindexing the tables at the same time the data_engine is bulk-inserting new records into the tables.

For example, the bulk insert contains a row for a insert on a locked object (being reindexed). The data_engine detects this, and saves all of the existing bulk insert packages. It then reinitializes the connection to the database (to make sure there isn't a problem), then inserts the saved bulk packages, and continues processing the data. So no data are being lost in this process. 

The data_engine may generated an alarm that says ""Bulk load data was expected but not sent. The batch will be terminated." This alarm is coming from the ODBC driver that the probe uses when attempting to create a rowset which is blocked for similar reasons.

When the data_engine generates either of these alarms, the commit fails and the data_engine restores the records back to the unprocessed queue. These records will be processed in the next cycle.

This is behavior by design, and has been implemented to catch the several hundred error conditions which can be generated from the ADO/Native Client layer, if something goes wrong

This alarm can also occur even if automatic reindexing is disabled in the data_engine probe's configuration if partitioning is enabled. The data_engine maintenance job creates new partitions and re-indexing will be performed during partition creation.

Additional Information

By default, these alarms are issued with severity = critical. There is an option to set the alarm_severity flag in the raw-configure module of the data_engine to raise these alarms with a different alarm level if that's desirable. But the drawback is of course that you might get into a situation where the ADO/Native Client layer generates a generic error message which you might want to give attention. Unfortunately we're unable to control the messages provided by these layers, so we just present them when they come.