Minions quit responding to commands after a while

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Executing test.ping in the Aria Config (raas) GUI did not show a result in the Activity page.
• Real production salt jobs coming from vRA/vRO to raas were not completing successfully.

Environment

SaltProject 3006.13+

Resolution

Salt is a modular system written in Python making it very capable of being manually patched. 

One solution to this issue might be as follows. 

• Run salt --versions to confirm that your Salt version is greater than 3006.13
• Stop the Salt master
  • systemctl stop salt-master
• Download the ipc.py from the SaltProject Github repository.
  • https://github.com/saltstack/salt/blob/v3006.12/salt/transport/ipc.py
    • You will need to find the "Raw" button to download the file directly
    • Be sure to download the 3006.12 version of the file
• Place the ipc.py file on the Salt master and place it in /opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/ipc.py
  • This will overwrite the existing file which is expected
• Start the Salt master
• Wait for minions to reconnect to Salt master
  • You can generally confirm that minions are reconnected by one of the following
    • Run a "salt \* test.ping" from the Salt master command line interface
      • You should see a bunch of responses in green
    • Confirm that netstat -anp --tcp | grep -i 4505 returns the expected number of minion connections (should roughly match the number of accepted minion keys)
• Run a test command from the Aria Config UI and confirm that jobs are returning as expected.