Push failed messages in the messages file

Products

• Security Analytics

Issue/Introduction

/var/log/messages may report the following error messages:

2021-07-25T19:06:38+00:00 host_name shaft[30467]: insert_meta_record[1271]: push failed 10

2021-07-25T19:06:43+00:00 host_name shaft: Last message 'insert_meta_record[1' repeated 116695 times, suppressed by syslog-ng on host_name

Cause

This is caused when there are two VPNs or two routers talking between each other or the traffic is behind a NAT.  The indexing is not able to keep up.

Resolution

The very narrow flow variety prevented the best multi-threading.  The tuning allowed more threads in the indexing processes.  The /etc/sysconfig/solera-shaft configuration file needs to be updated. 

• Add a -N to the line beginning with SHAFT_OPTIONS
• Restart shaft with:   systemctl solera-shaft restart.  

This should be done on each sensor.

If this has already been done and the problem persists, then the capture rate needs to be decreased or additional hardware needs to be allocated.

Additional Information

The messages are indicating that your traffic is so similar that the algorithm to multithread the indexing of the packets creates fewer threads than normal.  With more threads, more processors can be indexing the packets.

This is called "simple NAT mode."