Salt master cross-calling modules after upgrade

Products

• VMware Aria Suite

Issue/Introduction

In some cases, it may occur that the Salt master ends up calling modules from a previous version of Salt. This can be identified by running any command from the CLI of the Salt master such as, salt \* test.ping, and confirming whether you see warnings related to a previous version of Python, ie Python 3.6. 

This can be further confirmed by checking the /etc/salt/master.d/raas.conf or /etc/salt/master.d/eAPIMasterPaths.conf file for references to /usr/lib/pythonX.X. 

Environment

Salt 3006+

Cause

The upgrade was not performed cleanly, or the master plugin was not reinstalled properly with an updated configuration pointing to the new correct locations for additional modules.

Resolution

You will need to remove any old versions of Salt from other Python environments that may be running and update the master plugin configuration to point to the correct locations for modules.

1. Stop the Salt master
2. Backup the Salt master configuration
   1. This is just in case any incorrect changes are made
3. Run find /usr -type d -iname "salt" to find old directories and remove them
   1. If you are running 3006.x , then Salt should be running from /opt/saltstack/salt. Leave this directory in tact. 
   2. Other directories, such as /var/run/salt or /var/cache/salt are to be left in tact unless you plan to fully reinstall Salt
4. Remove any salt directories found under the /usr/ directory
5. Run /opt/saltstack/salt/extras-3.10/bin/sseapi-config --ext-modules > /etc/salt/master.d/eAPIMasterPaths.conf
6. Check the /etc/salt/master.d/raas.conf file for any references to module directories and remove these references
   1. Some examples are states_dirs, returner_dirs,  and engines_dirs
   2. You can also compare to the newly generated file. Any option listed in eAPIMasterPaths.conf should be removed from all other files.
   3. The sseapi-config command may also be found in /opt/saltstack/salt/bin/sseapi-config depending on installation method
7. Run /opt/saltstack/salt/extras-3.10/bin/sseapi-config --default-config > /root/raas.conf to generate a new master plugin configuration
   1. You will need to update this configuration per the documentation https://docs.vmware.com/en/VMware-Aria-Automation/8.16/Installing-Automation-Config/GUID-E5ECC4A0-ABEF-475D-8BF9-53429C0CB6DB.html
8. Depending on the version of master plugin, you may need to remove /etc/salt/pki/master/raas_key.pub and /etc/salt/pki/master/sseapi_key.pub before restarting the Salt master
   1. New keys will be generated on startup of the Salt master
9. Restart the Salt master
10. Verify that the Salt master is operating as expected by running salt \* test.ping 
    1. There should be no warnings related to other Python environments