Aria Config jobs show results from minions not targeted
Article ID: 388015
Updated On:
Products
Issue/Introduction
Job results targeting a grain based target, may result in additional minions responding to the job. Usually the result will show that the minion didn't respond. This results in a series of events on the Salt master event bus like the series of events listed below.
20###############88 {"minions": ["normal_minion", "unresponsive"], "_stamp": "2025-02-11T19:44:48.480580"}
salt/job/20###############88/new {"jid": "20###############88", "tgt_type": "compound", "tgt": "G@os_family:Debian", "user": "root", "fun": "state.apply", "arg": ["sample_state"], "minions": ["normal_minion", "unresponsive"], "missing": [], "_stamp": "2025-02-11T19:44:48.481374"}
20###############45 {"minions": ["unresponsive"], "_stamp": "2025-02-11T19:44:48.491034"}
salt/job/20###############45/new {"jid": "20###############45", "tgt_type": "list", "tgt": ["unresponsive"], "user": "root", "fun": "saltutil.find_job", "arg": ["20###############71"], "minions": ["unresponsive"], "missing": [], "_stamp": "2025-02-11T19:44:48.491709"}
salt/raas_master/iteration {"metrics": {"raas_master_commands_processed": 1, "raas_master_sseapi_engine_iteration_seconds": 0.06570172309875488}, "_stamp": "2025-02-11T19:44:48.501409"}
minion/refresh/normal_minion {"Minion data cache refresh": "normal_minion", "_stamp": "2025-02-11T19:44:48.715651"}
salt/job/20###############88/prog/normal_minion/0 {"id": "normal_minion", "tag": "salt/job/20###############88/prog/normal_minion/0", "data": {"ret": {"name": "test.sleep", "changes": {"ret": true}, "comment": "Module function test.sleep executed", "result": true, "__sls__": "sample_state", "__run_num__": 0, "start_time": "19:44:50.994207", "duration": 1004.086, "__id__": "go_to_sleep_0"}, "len": 3}, "cmd": "_minion_event", "jid": "20###############88", "_stamp": "2025-02-11T19:44:52.008542"}
salt/job/20###############88/prog/normal_minion/1 {"id": "normal_minion", "tag": "salt/job/20###############88/prog/normal_minion/1", "data": {"ret": {"name": "test.sleep", "changes": {"ret": true}, "comment": "Module function test.sleep executed", "result": true, "__sls__": "sample_state", "__run_num__": 1, "start_time": "19:44:52.012664", "duration": 1003.413, "__id__": "go_to_sleep_1"}, "len": 3}, "cmd": "_minion_event", "jid": "20###############88", "_stamp": "2025-02-11T19:44:53.022410"}
salt/raas_master/iteration {"metrics": {"raas_master_sseapi_engine_iteration_seconds": 0.030799150466918945}, "_stamp": "2025-02-11T19:44:53.475928"}
salt/job/20###############88/prog/normal_minion/2 {"id": "normal_minion", "tag": "salt/job/20###############88/prog/normal_minion/2", "data": {"ret": {"name": "test.sleep", "changes": {"ret": true}, "comment": "Module function test.sleep executed", "result": true, "__sls__": "sample_state", "__run_num__": 2, "start_time": "19:44:53.026295", "duration": 1003.259, "__id__": "go_to_sleep_2"}, "len": 3}, "cmd": "_minion_event", "jid": "20###############88", "_stamp": "2025-02-11T19:44:54.037617"}
salt/job/20###############88/ret/normal_minion {"cmd": "_return", "id": "normal_minion", "success": true, "return": {"module_|-go_to_sleep_0_|-test.sleep_|-run": {"name": "test.sleep", "changes": {"ret": true}, "comment": "Module function test.sleep executed", "result": true, "__sls__": "sample_state", "__run_num__": 0, "start_time": "19:44:50.994207", "duration": 1004.086, "__id__": "go_to_sleep_0"}, "module_|-go_to_sleep_1_|-test.sleep_|-run": {"name": "test.sleep", "changes": {"ret": true}, "comment": "Module function test.sleep executed", "result": true, "__sls__": "sample_state", "__run_num__": 1, "start_time": "19:44:52.012664", "duration": 1003.413, "__id__": "go_to_sleep_1"}, "module_|-go_to_sleep_2_|-test.sleep_|-run": {"name": "test.sleep", "changes": {"ret": true}, "comment": "Module function test.sleep executed", "result": true, "__sls__": "sample_state", "__run_num__": 2, "start_time": "19:44:53.026295", "duration": 1003.259, "__id__": "go_to_sleep_2"}}, "retcode": 0, "jid": "20###############88", "fun": "state.apply", "fun_args": ["sample_state"], "user": "root", "out": "highstate", "_stamp": "2025-02-11T19:44:54.049942"}
salt/job/20###############88/returning {"_stamp": "2025-02-11T19:44:54.053575"}
What we see in the series of events above is that the Salt master publishes the original job targeting the one expected minion along with an additional minion that is not currently responding and the Salt master doesn't have any cache information for.
Environment
Aria Config - all versions
Cause
The root cause of this issue is that the Salt master does not have any grains cached for this minion, so it can not be sure that it should not target the minion. The Salt master errs on the side of caution and publishes the job for both minions, the known match, and the unknown match. This is fine since the minion will also make a determination about whether it should execute the job. In the case above, the minion is not online, and doesn't receive the job anyway.
Resolution
The ideal solution for this is to ensure that minions are connecting properly to the Salt master. and that minion keys are in an accepted state. This will allow the master to update its minion cache as needed.
If this is feasible in your environment, then a possible workaround is to add a "*" (glob) target to the end of your grain based target to scope the target to known keys.
Feedback
