Background Worker Options

Products

• VMware Aria Suite

Issue/Introduction

Applicable versions: SaltStack Enterprise version 5.5 through 6.2

Overview

SaltStack Enterprise includes new background worker options to optimize performance for a range of deployment scenarios, since version 5.5. Background workers handle processes that run in the background, such as running jobs according to a given schedule, or syncing with an LDAP server.

New options for background workers are particularly useful for high-availability deployments. You can customize background worker settings to share background processes across multiple servers, as demonstrated in the following diagram.

 

How to configure options for background workers

To define background worker options, modify the raas configuration file when you install or upgrade SaltStack Enterprise, or any time you want to improve the performance of your Enterprise deployment.

See the Enterprise download and installation page for full installation instructions.

See Commands for a list of commands that are useful for troubleshooting.

 

Background Worker options

The following table describes available background worker options.

For recommended background worker settings, see Examples of background worker settings.

Option	Description
combined_process	Boolean to define whether both web server processes and background workers run on the server. If True, both types of processes run on the server. If False, only one type of process runs on the server.
concurrency	Number of worker processes. One unit represents one CPU core. If 0, automatically calculate one process per core, divided evenly between web server and background worker processes.
max_tasks	Number of tasks a worker can complete until it restarts.
max_memory	Amount of memory in kB a worker can consume until it restarts. If 0, define automatically based on available memory. If None, allow the maximum memory consumption possible, up to 1GB.

 

Commands

Use the following commands to assist with troubleshooting. The -l flag indicates debug mode.

Command	Description
raas -l info	Run both web server processes and background worker processes. To run only web server processes, add the following to etc/raas/raas before running the command. background_workers: combined_process: False
raas -l info worker	Run only worker processes.
raas -l info beat	Run only the beat process. The beat process initiates other processes at a set interval, distributing periodic tasks to workers.
raas celery purge	Purge the task queue.

 

Examples of background worker settings

You can customize background worker settings to improve performance for a range of deployment scenarios. Multiple scenarios are described below, with recommended settings for each.

Note: In the following examples, num_processes represents the number of web server processes.

Scenario 1: High availability

The following example is a shared system where both web processes and background worker processes run concurrently on each server, supported by a load balancer. Each server includes 10 CPU cores. Processing power is distributed evenly across both types of processes.

/etc/raas/raas

On each server:

num_processes: 5
background_workers:
 combined_process: True
 concurrency: 5
 max_tasks: 100000
 max_memory: 1048576

Scenario 2: Scaling web processes

The following example is a deployment that experiences a large amount of web server processes and relatively few background processes.

This scenario might be found, for example, in an environment that has many minions, and does not rely heavily on schedules. 

/etc/raas/raas

On one server:

num_processes: 5
background_workers:
 combined_process: True
 concurrency: 5
 max_tasks: 100000
 max_memory: 1048576

On a second server:

num_processes: 10
background_workers:
 combined_process: False

Scenario 3: Scaling schedules

The following example is a deployment that needs to support a large amount of background processes, and relatively few web server processes.

This scenario might be found, for example, in an environment that relies heavily on schedules. Each server includes 10 CPU cores.

/etc/raas/raas

On one server:

num_processes: 5
background_workers:
 combined_process: True
 concurrency: 5
 max_tasks: 100000
 max_memory: 1048576

On a second server:

background_workers:
 combined_process: False
 concurrency: 10

Scenario 4: Scaling schedules

The following example is a multi-node system where web processes and background worker processes run on separate servers. Each server includes 10 CPU cores.

This configuration provides a simple framework for scaling SaltStack Enterprise without a load balancer, by moving your background work to a second server.

/etc/raas/raas

On one server:

num_processes: 10
background_workers:
 combined_process: False

On a second server:

background_workers:
 combined_process: False
 concurrency: 10

 

Environment

VMware Aria Automation Config 8.12.x