Salt Minion Not Connecting to Master After Downtime

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Salt minion does not respond to commands after the master is brought online.

• Running salt-call test.ping on the minion returns no response.

• Minion logs show repeated reconnection attempts or inactivity.

Environment

Aria Automation Config 8.x

Cause

sudo salt-call --local config.get recon_default
sudo salt-call --local config.get recon_max
sudo salt-call --local config.get tcp_keepalive

Salt minions maintain persistent ZeroMQ connections to the master. When the master is unavailable for a prolonged period, the minion may:

• Fail to retry the connection due to default reconnect intervals.

• Hold a stale authentication key.

• Be impacted by firewall, DNS, or network issues.

Default recon_* parameters are not optimized for extended outages, causing the minion to stop attempting reconnection.

Resolution

Step 1: Verify Network Connectivity

1. From the minion, confirm connectivity to the master:

   ping <master-ip>
   nc -vz <master-ip> 4505
   nc -vz <master-ip> 4506

2. Resolve any DNS, firewall, or routing issues if connectivity fails.

Step 2: Restart Minion and Re-authenticate

1. Stop the minion service: sudo systemctl stop salt-minion

2. Remove the stale key file: sudo rm -f /etc/salt/pki/minion/minion_master.pub

3. Start the minion service: sudo systemctl start salt-minion

Step 3: Accept Keys on the Master

1. On the master, list and accept keys: 

   sudo salt-key -L
   sudo salt-key -A -y

Step 4: Test the Connection

1. From the master: sudo salt '*' test.ping

2. From the minion: sudo salt-call test.ping

Expected output: True

Step 5: Tune Reconnection Settings

1. Edit /etc/salt/minion and add/update:

   recon_default: 1000 # 1 second
   recon_max: 5000 # 5 seconds
   tcp_keepalive: True

2. Restart the minion to apply changes: sudo systemctl restart salt-minion

Verification

To confirm the settings are enforced:

These should return 1000, 5000, and True.