ALERT: Some images may not load properly within the Knowledge Base Article. If you see a broken image, please right-click and select 'Open image in a new tab'. We apologize for this inconvenience.

Excessive RAM utilization in shaft service

book

Article ID: 236845

calendar_today

Updated On:

Products

Security Analytics Security Analytics - VA

Issue/Introduction

The shaft service or shaft process is using more memory than expected. 

Environment

Release : 8.2.4-55248

Resolution

The purpose of the shaft process is to accumulate the packet metadata and save them as indexes. That being said, indexing will be dropped if there is more traffic to capture than can be indexed.  The indexing of flows is a second priority to the capturing of the packets.

Shaft will store the metadata to a memory based database table.  If there are flows which are long living, like smtp traffic or proxy traffic between servers, there is a large amount of data to store and the memory usage will need to increase to accommodate. The longer the flow, the larger the table.

The shaft table will also grow larger if there are many, many flows which may not be large but may have long lives.  Each flow must be stored until the timeout is reached to efficiently record the indexes.  Once the flow times out or closes, then that entry can be flushed to disk. 

If the system is capturing at an extremely high rate, for example over 4.5Gb/s, then shaft may lose track of the flows and miss the SYN/ACK and hold memory for that flow until it times out.  There may be just too many flows where it thinks they may still be active but have actually closed.

To mitigate, you can restart shaft.  If there is a memory leak in shaft then this would clear out the unused memory.  But, when you restart shaft, you lose all of the indexing of flows which were stored.  As of the writing of this article (March 2022) we are unaware of any memory leaks in shaft, at this time.

To restart shaft, you can run as root from the cli:  systemctl restart solera-shaft