High swap usage alarms on Salt Master due to pillar rendering failure

Products

• VMware Salt

Issue/Introduction

Salt Master servers may experience continuous swap usage alarms and high memory consumption. This issue typically occurs when pillar data fails to render correctly, causing jobs to back up and exhaust system resources. This is often linked to stale or decommissioned server references within the configuration.

• The threshold for the Swap Space Available MBytes performance counter is exceeded.
• Available MBytes drops to critical levels (e.g., 2 MB).
• Salt Master logs show errors related to rendering pillar data.
• High memory usage by Salt processes even after disabling the GUI.

Environment

• VMware Salt 3006.x
• Aria Automation Config / SaltStack Config 8.x

Cause

Pillar data fails to render because decommissioned or unreachable servers are still referenced in the sse_settings.yaml file. When the Salt Master attempts to render pillar data for these missing references, the process fails or hangs, leading to a backlog of jobs that consume system memory and swap space.

Resolution

To resolve this issue, remove stale references and refresh the Salt cache and pillar data.

1. Edit the sse_settings.yaml file and remove any references to decommissioned or retired servers.

2. Force the active consolidated minion to update its grains cache:

   salt '####' saltutil.refresh_grains

3. Remove the cached pillar and grains data for the decommissioned minion:

   salt-run cache.clear_all ####

4. Refresh the in-memory pillar data for the consolidated node:

   salt '####' saltutil.refresh_pillar wait=True

Additional Information

• If the issue persists, review the Salt Master logs for specific Jinja2 rendering errors.
• For instructions on creating support cases, see Creating and Managing Broadcom Support Cases.