How to upgrade salt minions from the salt master or Salt your Salt or Upgrade minions at scale or minions timeout and fail to start after upgrade

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

This article provides a sample state to update windows salt minions in bulk from the salt master, so it is not necessary to log onto each minion host and do it by hand.

• User's own state may cause timeouts on the minion when attempting to use SLS files (Salt states) to upgrade the minion itself
• Manually logging in to each minion and running the upgrade is not feasible at scale
•  

Environment

• Salt-master: all versions
• Salt-minion: all versions
• Aria Automation Config: all versions
• VMware Aria Automation Config (formerly SaltStack Config)
• Salt Versions: 3006.x, 3007.x, and newer.
• Operating Systems:
  • RedHat Enterprise Linux (RHEL) 8 and 9 (and derivatives like AlmaLinux, Rocky Linux).
  • Microsoft Windows Server (2016, 2019, 2022).

Cause

If installing on Linux, then RPM or Deb package might be restarting the minion daemon too early causing the job to not return to the Salt master. This in turn leads to the minion's connection to the Salt master being lost. 

If installing on Windows, a similar issue may occur but is more likely due to the installer running too soon not allowing the job to return before beginning the upgrade process.

Resolution

NOTE: We strongly recommend that you test this SLS file in a lab prior to executing this in your production environment. This is only intended as sample code.

The SLS file provided below is not intended to cover every use case or scenario for being able to manage the deployed minion version, but is intended to serve as an example of how this might be done at scale.

Step 1: Create the Management State File

On your Salt Master, create a new file (e.g., /srv/salt/os_updates/manage-salt.sls) and paste the following content:

{% set target_version = salt['pillar.get']('target_version', 'latest') %}

# Define variables
{% if grains['os_family'] == 'RedHat' %}
  # RHEL: Manage both packages to avoid dependency locks
  {% set packages_to_manage = ['salt', 'salt-minion'] %}
  {% set service_name = 'salt-minion' %}

  {% set restart_cmd = 'nohup sh -c "sleep 15; systemctl restart salt-minion" > /dev/null 2>&1 &' %}
  {% set pre_mask = 'systemctl mask salt-minion' %}
  {% set post_unmask = 'systemctl unmask salt-minion' %}

  # LOGIC SWITCH: If we are on RHEL and targeting a specific version (downgrade/pin),
  # we switch to 'cmd.run' to bypass Salt's pkg module dependency issues.
  {% if target_version != 'latest' %}
    {% set install_type = 'rhel_cmd' %}
  {% else %}
    {% set install_type = 'pkg' %}
  {% endif %}

{% elif grains['os_family'] == 'Windows' %}
  # --- WINDOWS INSTALLER DETECTION ---
  # 1. Check for custom grain 'installer_type'
  {% set installer_type = salt['grains.get']('installer_type') %}

  # 2. Fallback: If grain is missing, detect via Registry
  {% if not installer_type %}
    # PowerShell to get the UninstallString. MSI uses MsiExec.exe; EXE uses uninst.exe.
    # We use cmd.run during Jinja rendering to dynamically set this variable.
    {% set ps_check = "Get-ChildItem HKLM:\\Software\\Microsoft\\Windows\\CurrentVersion\\Uninstall | Get-ItemProperty | Where-Object {$_.DisplayName -like 'Salt Minion*'} | Select-Object -ExpandProperty UninstallString" %}
    {% set uninstall_string = salt['cmd.run'](ps_check, shell='powershell') %}

    {% if 'MsiExec.exe' in uninstall_string %}
      {% set installer_type = 'msi' %}
    {% else %}
      {% set installer_type = 'exe' %}
    {% endif %}
  {% endif %}

  # 3. Set Package Name based on Type
  # These names correspond to standard Salt WinRepo definitions
  {% if installer_type == 'msi' %}
    {% set packages_to_manage = ['salt-minion-msi-py3'] %}
  {% else %}
    {% set packages_to_manage = ['salt-minion-py3'] %}
  {% endif %}
  # -----------------------------------

  {% set service_name = 'salt-minion' %}
  {% set restart_cmd = 'start /b powershell -command "Start-Sleep -Seconds 30; Restart-Service -Name salt-minion -Force"' %}

  # WINDOWS PINNING:
  # If 'latest' is requested, force pin to 3006.17 to avoid upgrading to 3007.x branch.
  {% if target_version == 'latest' %}
    {% set target_version = '3006.17' %}
  {% endif %}

  # WINDOWS WORKAROUND:
  # Use 'module.run' to force execution, bypassing state idempotency issues during downgrades.
  {% set install_type = 'win_module' %}

{% endif %}

# -- Execution Block --

{% if grains['os_family'] in ['RedHat', 'Windows'] %}

# [RHEL ONLY] Mask service
{% if grains['os_family'] == 'RedHat' %}
mask_service_to_prevent_rpm_restart:
  cmd.run:
    - name: {{ pre_mask }}
    - unless: systemctl is-enabled {{ service_name }} | grep masked
    - require_in:
      - {{ install_type }}: install_or_downgrade_minion
{% endif %}

# [ALL] Perform the Package Upgrade/Downgrade
install_or_downgrade_minion:
  {% if install_type == 'rhel_cmd' %}
  # RHEL WORKAROUND: Raw DNF command
  cmd.run:
    - name: dnf install -y {% for pkg in packages_to_manage %}{{ pkg }}-{{ target_version }} {% endfor %}
    - unless: rpm -q {% for pkg in packages_to_manage %}{{ pkg }}-{{ target_version }} {% endfor %}

  {% elif install_type == 'win_module' %}
  # WINDOWS WORKAROUND: Force install via execution module
  module.run:
    - pkg.install:
      - name: {{ packages_to_manage[0] }}
      - version: {{ target_version }}
    # Check if we are already at the correct version to preserve idempotency
    - unless:
      - cmd: 'salt-call --local pkg.version {{ packages_to_manage[0] }} | findstr /C:"{{ target_version }}"'

  {% else %}
  # Standard Logic (RHEL Upgrades)
    {% if target_version == 'latest' %}
  pkg.latest:
    - pkgs: {{ packages_to_manage }}
    {% else %}
  pkg.installed:
    - pkgs:
      {% for pkg in packages_to_manage %}
      - {{ pkg }}: {{ target_version }}
      {% endfor %}
    - allow_updates: True
    - allow_downgrade: True
    - ignore_epoch: True
    {% endif %}
    - refresh: True
  {% endif %}

# [RHEL ONLY] Unmask service
{% if grains['os_family'] == 'RedHat' %}
unmask_service_after_install:
  cmd.run:
    - name: {{ post_unmask }}
    - require:
      - {{ install_type }}: install_or_downgrade_minion
{% endif %}

# [ALL] Trigger Delayed Restart
{% if grains['os_family'] == 'RedHat' %}
schedule_delayed_restart:
  cmd.run:
    - name: {{ restart_cmd }}
    - bg: True
    - onchanges:
      - install_or_downgrade_minion
{% else %}
schedule_delayed_restart:
  cmd.run:
    - name: schtasks /create /tn "Salt-Minion-Restart" /tr "powershell.exe -Command Restart-Service -Name salt-minion -Force" /sc once /st (Get-Date).AddMinutes(1).ToString("HH:mm") /f /rl highest
    - bg: True
    - onchanges:
      - install_or_downgrade_minion
{% endif %}
{% endif %}

Step 2: Running the State

Scenario A: Upgrade to the "Safe" Latest Version

By default, running this state without Pillar data will upgrade RHEL systems to the absolute latest version available in the repo, and Windows systems to the pinned "safe" version defined in the SLS file (default 3006.17).

salt 'target-minion' state.apply os_updates.manage-salt

Scenario B: Downgrade or Pin to a Specific Version

To force a specific version (upgrade or downgrade), pass the target_version via Pillar.

For RHEL (include release if needed):

salt 'rhel-minion' state.apply os_updates.manage-salt pillar='{"target_version": "3006.17-0"}'

For Windows (version string only):

salt 'windows-minion' state.apply os_updates.manage-salt pillar='{"target_version": "3006.17"}'

Additional Information

NOTE: This article assumes that winrepo has been configured. See the WinRepo module documentation for more info.

NOTE: If using this state as a job in the UI, {"package": "salt-minion-py3"} will need to be added in the "Pillar override" section so that it can find the package.

NOTE: The job may not return or may return as failed due to the required minion service restart.

You can verify the version directly on the server that was upgraded, or you can run the following from the master to verify the version of the salt-minion:

salt minion-id cmd.run 'salt-call --versions'

Troubleshooting

• "Already at expected version": If this occurs during a downgrade, ensure you are using the provided SLS which utilizes cmd.run (RHEL) or module.run (Windows) to bypass state-level version checks.
• Service Deadlocks: If the minion goes offline and never returns, ensure the mask_service_to_prevent_rpm_restart step is succeeding. This prevents the RPM from restarting the service while the connection is active.