Salt state performance is significantly slower when the master is connected to VCF Salt RaaS than when it is not

Products

• VMware Salt

Issue/Introduction

• Running SLS files on minions takes much longer (e.g. minutes rather than seconds) when the master is configured to connect to the sseapi RaaS server backend, compared to when the roots salt-master backend is used first
  • The performance difference depends on how many state files are searched for, and whether they can be found on the first-searched fileserver backend
• There are many errors for fs.get_file requests in the master log  (with debug logging enabled):
  • [sseapiclient.httpclient:885 ][DEBUG   ][####] Sending POST request (id #####, resource.method fs.get_file, riq 1501, jid none) to https://<RaaS-FQDN>/rpc
[salt.loaded.ext.fileserver.sseapi_fs:105 ][DEBUG   ][####] Failed to get file(path: <SLS-Path>; saltenv: <Salt-Env>): No file matching '<SLS-Path>' was found in the '<Salt-Env>' saltenv
• The fileserver_backend setting in the master config puts RaaS (sseapi) before Master FS (roots):
  • fileserver_backend:
  - sseapi
  - roots

Environment

• Salt open 3006.x
• VCF Salt 8.18.3 or earlier

Cause

The state files needed are searched on the first fileserver_backend, which must fail first before the correct backend is searched.

These failures create the performance difference. It is exacerbated where each minion searches for multiple state files.

The same issue could exist in the reverse direction: if the SLS files are kept on the RaaS filesystem, but the Master FS (roots) is configured as the first fileserver_backend.

Resolution

This is a known issue and an optimization is scheduled for inclusion in a later release of Salt.

A possible workaround is to search for SLS files in the master filesystem before searching on the RaaS fileserver:

In the master config file:

fileserver_backend:
  - roots
  - sseapi

 

If the issue is experienced in the reverse direction, then the reverse workaround should be used, placing sseapi backend before roots.