Benchmarking Guide For SaltStack Enterprise

Products

• VMware Aria Suite

Issue/Introduction

A guide for benchmarking to help size your systems properly.

Introduction 

This guide was created to help customers of SaltStack Enterprise (SSE) benchmark their systems in a standard fashion that can be compared to a reference SSE system. This will help the SaltStack support staff recommend upgrades and troubleshoot performance bottlenecks.

For ease of use and setup, the Phoronix Test Suite is used to run selected benchmarks. You can download Phoronix from: https://www.phoronix-test-suite.com/?k=downloads

For network testing iperf3 will also need to be installed. This utility is usually available in your OS package manager.

When performing the benchmarks, you don’t necessarily have to run them on your production environment. You should run them on a machine that will emulate the same specs as the component you plan to run on it.

Note on large shared systems: If your system is large and doesn’t have dedicated RAM or CPUs, please consult your support contact. Some benchmarks may overuse the RAM or CPU, causing long benchmark times and conflicts with other devices sharing resources.

Reference system 

The reference system below is not necessarily recommended for every piece of SSE infrastructure but only used for a baseline comparison. For example, your Redis server can often have lower specs and a slower disk. Many times you’ll want a faster system for your Postgres DB if you are operating at a larger scale. SaltStack Customers have a variety of hardware and these benchmarks help determine the ability of your systems compared to a baseline. System recommendations will be made by support staff based on each client's results and based on the desired scale and use cases.

Reference System Specs:
Amazon AWS: m5d.2xlarge
Cores: 8
Memory: 32GB
Disk: 1 x 300 NVMe SSD
Bandwidth: Up to 10Gbps

SSE server benchmarks 

Run the following benchmarks on a machine that is similar to the machine you will run the raas service on. The raas service runs a web server with background workers.
 

Benchmark	Test	Reference System
nginx	---	Avg: 27,224.89 Requests Per Second
pybench	---	Avg: 1370 Milliseconds
sysbench	CPU	Avg: 7,307.63 Events Per Second
sysbench	Memory	Avg: 8,908,310.27 Events Per Second

RaaS server to Postgres network benchmark 

To benchmark networking, install iperf3 on both the Postgres Server and RaaS Server. 

1. On the Postgres Server run: iperf3 -s 
2. On the RaaS Server run: iperf3 -c {pg-ip-address}

Reference System Results: 9.57 Gbits/sec
 

RaaS server to Redis network benchmark 

To benchmark networking, install iperf3 on both the Redis Server and RaaS Server.

1. On the Redis Server run: iperf3 -s 
2. On the RaaS Server run: iperf3 -c {redis-ip-address}

Reference System Results: 9.57 Gbits/sec

 

Redis server benchmarks 

Run the following benchmarks on a machine that is similar to the machine on which you will run the Redis Service.

Benchmark	Test	Reference System
redis	LPOP	Avg: 1,831,649.29 Requests Per Second
	SADD	Avg: 1,477,841.83 Requests Per Second
	LPUSH	Avg: 1,208,712.50 Requests Per Second
	GET	Avg: 1,763,672.13 Requests Per Second
	SET	Avg: 1,342,488.21 Requests Per Second

Postgres server benchmarks 

Run the following benchmarks on a machine that is similar to the machine on which you will run the Postgres Database.

Benchmark	Test	Reference System
pgbench Note: this test takes a long time, usually at least 1hr+	Scaling: On-Disk Test: Single Thread Mode: Read Only	Avg: 23379.73 TPS
	Scaling: On-Disk Test: Single Thread Mode: Read Write	Avg: 96.00 TPS
compress-pbzip2	---	Avg: 9.66 Seconds
sysbench	CPU	Avg: 7,307.63 Events Per Second
sysbench	Memory	Avg: 8,908,310.27 Events Per Second

Salt Master benchmarks 

Run the following benchmarks on a machine that is similar to the machine on which you will run your Salt Master.

Benchmarks	Test	Reference System
pybench	---	Avg: 1370 Milliseconds
iozone	Record Size: 64Kb File Size: 512MB Disk Test: Read Performance	Avg: 6155.72 MB/s
iozon	Record Size: 64Kb File Size: 512MB Disk Test: Write Performance	Avg: 493.56MB/s
compress-pbzip2	---	Avg: 9.66 Seconds
sysbench	CPU	Avg: 7,307.63 Events Per Second
sysbench	Memory	Avg: 8,908,310.27 Events Per Second

Salt Master to RaaS Server network benchmark

To benchmark networking, install iperf3 on both the Salt Master and RaaS Server.

1. On the RaaS Server run: iperf3 -s 
2. On the Salt Master run: iperf3 -c {raas-ip-address}

Reference System Results: 9.57 Gbits/sec

Environment

VMware Aria Automation Config 8.12.x