Unable to run jobs from Salt WebUI for Salt Minions managed by two separate Salt Masters

Products

• VMware Tanzu Platform Services - SALT
• VCF Automation

Issue/Introduction

When attempting to run a job that targets minions across masters, master pairs you notice that not all minions return that are connected to one or the other master.

Environment

Aria Config 8.16.x+

Cause

The calculated targets have become stale, or corrupted due to configuration change, or a master communication to RaaS becoming stale. 

Resolution

1. Stop the problematic Salt master(s)

2. Stop the RaaS daemon

3. Login to the database

4. Run the query "delete from masters where master_id = '<PROBLEMATIC_MASTER_NAME>'; "

  a. Repeat step for as many masters as you need

5. On the Salt master, delete the raas_pub.key and the sseapi_pub.* from the /etc/salt/pki/master directory

6. Confirm that the Salt master plugin (raas.conf) is the latest version by running /opt/saltstack/salt/extras-3.10/bin/sseapi-config --default-config > /etc/salt/master.d/raas_new.conf

  a. The exact path for this command may vary depending on your version. Run `find / -type f -iname sseapi-config` if you are not able see it in the above path.

7. Edit the new configuration file ensuring that it uses the tgtmatch engine under the "engines" section of the configuration.

  a. If you use multiple files for your configuration, then you may need to run "salt-run test.get_opts --output=yaml" to confirm that the configuration appears as expected for all options

8. Start the RaaS daemons and confirm the UI is up by logging in

9. Start the Salt masters

10. Accept the key for the Salt master in the web UI

11. Run a test job from the UI targeting at least two minions that are not connected to the same master in the UI.