Push failed messages in the messages file

Products

• Security Analytics

Issue/Introduction

/var/log/messages may report the following error messages:

2021-07-25T19:06:38+00:00 host_name shaft[30467]: insert_meta_record[1271]: push failed 10

2021-07-25T19:06:43+00:00 host_name shaft: Last message 'insert_meta_record[1' repeated 116695 times, suppressed by syslog-ng on host_name

Cause

The is caused when the indexing algorithms can not keep up. This is an excessive rate.  This could be the overall volume of packets captured or this can also be caused when there are two VPNs or two routers talking between each other or the traffic is behind a NAT. 

Resolution

The typical cause is when the capture rate is excessive.  This solution is to reduce the number of packets captured.  

The other solution for a very narrow flow variety is to adjust for best multi-threading.  The tuning allows more threads in the indexing processes.  The /etc/sysconfig/solera-shaft configuration file needs to be updated with the algorithm changes.

Login as root to the command line and edit /etc/sysconfig/solera-shaft with your favorite editor.

• Add a -N to the line beginning with SHAFT_OPTIONS.  For example, 

  SHAFT_OPTIONS=-m 10 -N

• Save the file
  
  
• Restart shaft:   systemctl solera-shaft restart  

This should be done on each sensor.

If this has already been done and the problem persists, then the capture rate needs to be decreased or additional hardware needs to be allocated.

Additional Information

The messages are indicating that your traffic is so similar that the algorithm to multithread the indexing of the packets creates fewer threads than normal.  With more threads, more processors can be indexing the packets.

This is called "simple NAT mode."