Salt master fails to start with error "Exception caught while initializing gitfs remote"

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Salt master starts and then immediately crashes
• Typically occurs after upgrading the salt-master to a new version
• Master log at /var/log/salt/master reports the following error:
  • [CRITICAL] Exception caught while initializing gitfs remote 'ssh://<user>@<server>:<port>/path/to/repo.git': Repository not found at /var/cache/salt/master/gitfs/<cache_id>/_
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/gitfs.py", line 498, in __init__
    self.new = self.init_remote()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/gitfs.py", line 1920, in init_remote
    self.repo = pygit2.Repository(self._cachedir)
  File "/opt/saltstack/salt/extras-3.10/pygit2/repository.py", line 1606, in __init__
    path_backend = init_file_backend(path, flags)
_pygit2.GitError: Repository not found at /var/cache/salt/master/gitfs/<cache_id>/_
[CRITICAL] Failed to load gitfs
[CRITICAL] Master failed pre flight checks, exiting
• Navigating to /var/cache/salt/master/gitfs/<cache_id>/_ and running the command git status returns the following:
  • fatal: detected dubious ownership in repository at '<path to the repository>'
To add an exception for this directory, call:
git config --global --add safe.directory <path to the repository>

 

Environment

Salt master leveraging gitfs

Cause

Starting in git 2.35.2 users are prevented from executing git commands if the repository is owned by a different user

Resolution

1. Confirm the user that the salt-master is configured to run as. This is set with the 'user' key in the salt-master config. If this setting is not explicitly set then it will default to the root user. This is documented below:
   1. https://docs.saltproject.io/en/latest/ref/configuration/master.html#user
2. Check the ownership of the git repositories cached on the salt-master with the command below:
   1. ls -l /var/cache/salt/master/gitfs/*/_/
3. If the directories at /var/cache/salt/master/gitfs/*/_/.git are owned by a different user than the salt-master is configured to run as update the ownership. For example, if the salt-master runs as the root user but /var/cache/salt/master/gitfs/*/_/.git is owned by salt use the following command to correct the permissions:
   1. chown root:root /var/cache/salt/master/gitfs/*/_/.git
4. Restart the salt-master and monitor the logging to ensure that the service remains up and available