salt-minion services on a windows machine in paused state and do not write any logs.

Products

• VCF Operations/Automation (formerly VMware Aria Suite)
• VMware SaltStack

Issue/Introduction

On a Windows deployment of salt-minion, the salt-minion service is in pending status.
Minion logs are blank
re-installing minion does not help as the service is stuck in pending state

Environment

Windows Server 

Salt-open 3006.x
salt-minion 3006.x

Cause

Running salt-minion on debug mode will shows the below:

PS C:\>cd "C:\Program Files\Salt Project\Salt"
PS C:\Program Files\Salt Project\Salt> .\salt-minion.exe -l debug
Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\Libunpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Salt Project\Salt\Libunpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Salt Project\Salt\salt-minion.exe\__main__.py", line 4, in <module>
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\__init__.py", line 147, in <module>
    import salt._logging  # isort:skip
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\_logging\__init__.py", line 13, in <module>
    from salt._logging.impl import (
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\_logging\impl.py", line 506, in <module>
    import pip._internal.utils._log as pip_log_module  # pylint: disable=no-name-in-module,import-error
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\pip\_internal\__init__.py", line 3, in <module>
    from pip._internal.utils import _log
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 879, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1016, in get_code
  File "<frozen importlib._bootstrap_external>", line 1073, in get_data
PermissionError: [Errno 13] Permission denied: 'C:\\Program Files\\Salt Project\\Salt\\Lib\\site-packages\\pip\\_internal\\utils\\__init__.py'
PS C:\Program Files\Salt Project\Salt>

 

Using procmon to trace the process returns the below:

Based on above, AV solution (Trend Micro in the above case) was blocking the creation of the .py files. 

 

Resolution

Add 'C:\Program Files\Salt Project\Salt\Lib' to exclusion on the AV solution. .  Salt-minion is built on python and will attempt to create and execute the python files in the directory.