Salt master suffers poor system performance after when using salts python runtime as the OS default

Products

• VMware Aria Suite

Issue/Introduction

On RHEL machines the /usr/bin/python3 file is a symlink that points to /usr/bin/python3.9

If this symlink is updated to point to /opt/saltstack/salt/bin/python3.10 the overall performance of the salt-master machine will deteriorate

While this KB uses RHEL 9 as an example the issue can occur for any Linux based OS where similar changes are made

Environment

Any machine with salt installed via onedir (https://docs.saltproject.io/en/latest/topics/packaging/index.html) 

Cause

The OS does not expect to have all of its tooling run through the python environment provided by a salt installation for use by the salt installation and can cause unforeseen issues, including a significant degradation in OS performance 

Resolution

1. Do not modify symlinks provided by the OS vendor to use the python environment provided by the salt installation in /opt/saltstack
2. If the symlink has been modified use a command like the one below to restore it (this command assumes RHEL 9, your details may differ)
   1. ln -s /usr/bin/python3 /usr/bin/python3.9