"/var" directory is constantly full on Salt Master even after clearing the logs from directory

Products

• VMware
• VMware Aria Suite

Issue/Introduction

Symptoms:

• "/var" directory is constantly full on Salt Master even after clearing the logs from directory
• The "/var" directory fills up 100% every other day no matter how many times the logs are cleared
• The master logs under /var/log/salt/ consumes the maximum space and constantly inflates.
• If you tail the master logs file under /var/log/salt/ you will debug log entries :

[salt.loaded.ext.module.sseapi_local_queue:83 ][DEBUG  ][53998] queue sseapi-rpc: sql: SELECT COUNT(*) FROM queue

[salt.loaded.ext.module.sseapi_local_queue:80 ][DEBUG  ][53998] queue sseapi-rpc: closed database /var/cache/salt/master/sseapi-rpc.sqlite3

 [salt.loaded.ext.module.sseapi_local_queue:72 ][DEBUG  ][53998] queue sseapi-rpc: opening database /var/cache/salt/master/sseapi-rpc.sqlite3

 [salt.loaded.ext.module.sseapi_local_queue:83 ][DEBUG  ][53998] queue sseapi-rpc: sql: pragma journal_mode=wal

 

Environment

VMware Aria Automation Config 8.x
Salt 3005

Cause

• This happens if the logging level for the appliance is set to debug mode instead of warning or info.

Resolution

• The resolution is to set the correct logging level on the salt master appliance.
• Steps:

1. Take snapshot of the Salt master appliance.
2. Login to the salt master as root
3. Navigate to the /etc/salt/master.d directory:

cd /etc/salt/master.d

4. Open the master.conf file with vi editor and set the value for log_level_logfile from debug to warning:

vi master.conf

5. ensure the file has this entry log_level_logfile: warning
6. save the file
7. restart the salt-master service :

systemctl restart salt-master.service

Additional Information

Impact/Risks:

• If not set to correct logging level the /var directory will be filled 100% every now and then.