Jobs are not returned to raas from Salt master, showing "awaiting returns..." indefinitely in Aria Config GUI
Article ID: 404754
Updated On:
Products
Issue/Introduction
- Simple tasks such as test.ping are not returning for all masters in the Aria Config GUI:
- They show "awaiting returns..." and "0 of 0" (or 0 out of the total)
- The total number of masters may not reflect correctly here: for example if 2 masters are not working, it may say "4 of 4" even though there are 6 masters in total
- Jobs may fail when sent through certain masters, with the impacted masters changing with no clear pattern
Environment
- VMware Aria Config 8.x / Tanzu Guardrails for Workloads (formerly Saltstack Enterprise)
- Salt open 3006.11 or later
Cause
Salt 3006.11 introduced the ipc_write_timeout option which may be reached where the salt master takes a long time to return
Similar issues can be caused by missing engines from the master config, or outdated cache on the master - please see the Additional Information for this more general advice
Resolution
This issue will be resolved in the Salt 3006.15 release.
Workaround
Increasing the ipc_write_timeout value in salt master config to 60 seconds may alleviate the issue. For example, create /etc/salt/master.d/ipc.conf with:
ipc_write_timeout: 60
Additional Information
It is also important to make sure you follow the upgrade instructions on TechDocs carefully: the salt-master configuration should be re-generated after an upgrade and the new config tailored to your needs.
/opt/saltstack/salt/extras-3.10/bin/sseapi-config --default >/tmp/raas.conf- Enter master ID, raas URL, cluster ID and any other settings needed from the current config
- Replace the current config at
/etc/salt/master.d/raas.confwith this newline created and updated file
Refreshing the cache directory on the salt master can also help in some instances:
systemctl stop salt-mastermv /var/cache/salt/master ~/salt-cache-backupsystemctl start salt-master
Feedback
