Excessive RAM utilization in shaft service
Article ID: 236845
Updated On:
Products
Issue/Introduction
The shaft service or shaft process is using more memory than expected.
Environment
Release : 8.2.4-55248
Cause
If the system is capturing at an extremely high rate, for example over 4.0Gb/s, then shaft may lose track of the flows and miss the SYN/ACK and hold memory for that flow until it times out. There may be too many flows where it thinks they may still be active but have actually closed.
The packet capture may stop and reports may not return, also.
Resolution
The purpose of the shaft process is to accumulate the packet metadata and save them as indexes. That being said, indexing will be dropped if there is more traffic to capture than can be indexed. The indexing of flows is a second priority to the capturing of the packets.
Shaft will store the metadata to a memory based database table. If there are flows which are long living, like smtp traffic or proxy traffic between servers, there is a large amount of data to store and the memory usage will need to increase to accommodate. The longer the flow, the larger the table.
The shaft table will also grow larger if there are many, many flows which may not be large but may have long lives. Each flow must be stored until the timeout is reached to efficiently record the indexes. Once the flow times out or closes, then that entry can be flushed to disk.
To mitigate, you can restart shaft. If there is a memory leak in shaft then this would clear out the unused memory. But, when you restart shaft, you lose all of the indexing of flows which were stored. As of the writing of this article (March 2022) we are unaware of any memory leaks in shaft, at this time.
To restart shaft, you can run as root from the cli: systemctl restart solera-shaft.
We strongly advise that the capture rate be lowered below 4Gb/s. The capture rate can be viewed in the GUI in Capture -> Summary. Look for the value All in black in the graph in the top half of the page.
Feedback
