data_engine database connections and thread_count_insert setting

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

What is the best way to calculate or estimate the number of database connections I need to configure when I leverage the use of the data_engine's 'thread_count_insert' parameter?

Environment

Release: CNMSPP99000-8.4-Unified Infrastructure Mgmt-Server Pack-- On Prem
Component: data_engine

- UIM 8.0 and higher
- All supported databases

Resolution

The data_engine thread count relates to the threads for inserting high data rate RAW data. There are a number of other threads for other purposes and there is a thread pool.

This can be tested within your specific environment. The data engine opens as many database connections as the number of QOS definitions when it is the bulk insert mode, i.e., when the data_engine parameter, 'thread_count_insert' is set to a value larger than zero.

The data_engine creates one connection per QOS_DEFINITION, so you can determine the connections just from data_engine by executing the query:

select count(*) from S_QOS_DEFINITION

Also, UMP legacy UMP probes such as dap, dashboard_engine and wasp make direct connections to the database post-installation. In UIM 20.x, consider wasp and cabi.

Any other probes connecting to the same database server, will have connections of its own, e.g., sqlserver, mysql, db2, informix etc.

Add them all up to establish a good estimate of the total connection count.

***Note that when the thread_count_insert is set to 0, it should open only 1 connection due to disabling of any multi-threading/bulk inserts.

If you enable "bulk insert mode," by setting thread_count_insert to 1 or higher (the value does not matter in this regard), data_engine will make as many connections as there are QOS definitions.  There is no way to change this;  reducing the number of QoS definitions is not likely desirable because each one represents a unique QOS object a customer has chosen to monitor. 

As stated previously, setting thread_count_insert to 0 limits the data_engine to a single connection, but note that if you choose to set it to 0, the processing of QoS data will generally be much slower.

If you're on Windows, you can run a command like this to get a connection count for the data_engine:

 

>netstat -aB|find /i "data_engine" /c
307

 

Example for UNIX/Linux:

 

#ps -ef | grep -cw [d]ata_engine

220

Most customers have not reported any problems with the number of connections even when it reaches into the thousands.

Additional Information

MySQL-specific information

http://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html#sysvar_max_connections

and

http://dev.mysql.com/doc/refman/5.5/en/too-many-connections.html