Salt-minion agent has memory leak which may trigger OOM killer
search cancel

Salt-minion agent has memory leak which may trigger OOM killer

book

Article ID: 378306

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

Over time, because of a memory leak in the salt-minion, the amount of memory used by the salt-minion can increase to the point where the Linux OMM killer can start killing processes on the system leading to the system and the minion becoming inoperable. This may impact the minion's ability to execute new jobs.

 

 

Environment

Salt-minions 3006.5 Linux platform

Resolution

Upgrading your Salt minion to a version of 3006.8 but lower than 3007 (not inclusive 3007 at the original time of writing), should resolve this issue. However, in the event that an upgrade is not immediately possible, a potential workaround might to update the Systemd service file for the minion to limit the amount of memory it is permitted to use. 

  1. Edit the /usr/lib/systemd/system/salt-minion.service file
  2. Add the following values to the [Service] section of the file
    • MemoryHigh=2G
    • MemoryMax=3G
  3. Be sure to pick values that fit the needs and limitations of the host, or node running the salt-minion agent. 
  4. Run systemctl daemon-reload for the new values to take effect. 
  5. Restart the salt-minion agent with systemctl stop salt-minion && systemctl start salt-minion

For more information on configuring the salt-minion systemd settings https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html

Example configuration file

[Unit]
Description=The Salt Master Server
Documentation=man:salt-master(1) file:///usr/share/doc/salt/html/contents.html https://docs.saltproject.io/en/latest/contents.html
After=network.target

[Service]
MemoryHigh=1G #Specify the throttling limit on memory usage of the executed processes in this unit. Memory usage may go above the limit if unavoidable, but the processes are heavily slowed down and memory is taken away aggressively in such cases.
MemoryMax=2G # Specify the absolute limit on memory usage of the executed processes in this unit. If memory usage cannot be contained under the limit, out-of-memory killer is invoked inside the unit. It is recommended to use MemoryHigh= as the main control mechanism and use MemoryMax= as the last line of defense.
LimitNOFILE=100000
Type=notify
NotifyAccess=all
ExecStart=/usr/bin/salt-master

[Install]
WantedBy=multi-user.target

 

Powered by Wolken Software