Rotating Salt master keys periodically to comply business security practices

Products

• VMware Aria Suite

Issue/Introduction

Some businesses have a requirement to rotate any keys at a set interval for security. The problem is that if the key pair is changed on the Salt master and not updated on the Salt minions, the minions will refuse to process commands or communicate back to the Salt master since the public key does not match. Also, if using Salt masters in "active/active" mode, where minions connect to multiple masters, this presents an additional challenge since the key pair must be the same on both masters. The exception to this would be if multi-master PKI has been employed in the environment. 

Environment

SaltProject - all versions

Aria Config - all versions

Tanzu Salt - all versions

Resolution

There are multiple ways to solve this issue and the "best practice" for your organization may depend on exactly how your environment is deployed and what other services you may have access to. The below is a description of how this may be accomplished directly from Salt.

 

1. Create a backup of the existing keys
2. Generate the new key pair and save it to a place it in a temp directory
3. Copy the public key to the Salt master file server.
4. This is a public key intended to be shared, so we are not as worried about security here.
5. Go ahead and update the minion_master.pub file on the minions with the new public key in anticipation of the impending master restart
6. Move the keys into place on the Salt master
7. Restart the Salt master to use the new key pair
8. Restart the Salt minions so that they now also use the new key
9. Run a test.ping to confirm connectivity.
10. The orchestration may need to include a sleep or a wait_for_event to ensure that the minions come back successfully. This may depend on overall performance in your existing infrastructure. The orchestration that follows is a sample of what I mean and may not be 100% production ready. Please only use this as an example and make any changes necessary to match your environment.