Configuring Tanzu Salt with Master Failover

Products

• VMware Aria Suite

Issue/Introduction

This article describes how to configure Tanzu Salt with Salt Master Failover Servers.  

 

Environment

Aria Automation Config 8.x

Resolution

Salt supports a range of features for high availability and fault tolerance. For more on these features, see them listed alongside their configuration parameters in Configuration file examples.

 

Multimaster configurations

You can configure minions to connect to multiple masters in one of two ways:

• simultaneously - all masters are active, or "hot"
• failover - one master is active, but a second master will become active if the first becomes unavailable

 

Multimaster hot mode (default) 

To configure Salt Minions to connect to multiple masters at one time, configure the master parameter as a YAML list of all the available masters. By default, all masters are "hot", meaning that any master can direct commands to the Salt infrastructure.

In a multimaster configuration, each master must have the same cryptographic keys, and minion keys must be accepted on all masters separately. The contents of file_roots and pillar_roots must keep in sync with processes external to Salt as well.

Note: For a tutorial on setting up multimaster with "hot" masters, see Multimaster Tutorial.

 

 

Multimaster with failover 

To cause minions to connect to the first responding master in the list of masters, change the master_type parameter from str to failover. Every master_alive_interval seconds the minions will check to make sure the current master is still responding. If the master does not respond, the minion will attempt to connect to the next master in the list. If the minion runs out of masters, the list will be recycled in case dead masters have been restored.

Note that master_alive_interval must be present in the minion configuration, or else the recurring job to check master status will not get scheduled.

Note: Failover can be combined with PKI-style encrypted keys, but PKI is not required to use failover.

For more on Multimaster with PKI and Failover, see  Multi-master-PKI tutorial with failover.

You can combine master_type: failover with master_shuffle: True to spread minion connections across all masters (one master per minion, not each minion connecting to all masters). 

 

Prerequisites 

SaltStack Enterprise seamlessly integrates with a new or existing Salt installation. You must have a SaltStack Enterprise Server installed and connected to a Postgresql database. See SaltStack Installation Guide for more information.

 

Minion configuration 

Once installed, you need to set up Salt Minions for failover mode. To do this, set the following in your minion configuration.

• List masters in the desired order.

  master:
    - XXX.XXX.XXX.XXX
    - XXX.XXX.XXX.XXX
  

• Set the minion to failover type and define how often to check if the master is available. 

  master_type: failover
  master_alive_interval: 60

• Optionally set the minion to return to the first master in the list if it fails over to the secondary master, and define how often to check if the first master in the list is available.

  master_failback: True
  master_failback_interval: 30

• Optionally set the time to re-authenticate to prevent all minions from trying to reconnect at the same time. 

  random_reauth_delay: 60

For more on the above settings, see Configuring the Minion.

 Once you have configured the above, restart the salt-minion with the failover settings.

 

Master configuration 

Next, on each Master included in your failover cluster where you have installed the Enterprise Master Plugin, edit /etc/salt/master.d/raas.conf and configure the following settings to enable Failover support in the Enterprise Console (the GUI).  

• sseapi_failover_master: True
• sseapi_cluster_id: <name-of-failover-cluster-for-UI>

Restart the salt-master.The Enterprise Console (GUI) will now manage all minions in the failover cluster as a single group, sending the jobs to all of the masters although the minion is connected to only one master at a time.

Note: If you set up a HOT-HOT Multimaster scenario, you will set the sseapi_failover_master: False and set the sseapi_cluster_id.

 

Accepting minion keys on all masters 

To ensure minion keys are accepted on all masters, simulate the failure of all other masters so the minions send their keys to that specific master.

1. Stop master 2.

   [root@salt-master ~]# service salt-master stop
   

2. On master 1, wait for minions to connect and then accept minion keys.

   salt-key -A
   

3. Start master 2.

   service salt-master start
   

4. Stop master 1, then on master 2, wait for minions to connect and then accept minion keys.
5. Repeat the above steps, accepting keys on each master in sequence while all other masters are stopped.

Known issues 

• salt-run jobs will execute on every master that is part of the failover cluster.
• If any HOT-HOT minion is not connected to every master in the HOT-HOT cluster, there is a chance the minion will not receive the assigned job as the Enterprise server will send the job to only one master in the cluster.

 

Additional Information

The procedure is also applicable to the former names of this product, Saltstack Enterprise Config and Automation Config.