salt-master 401 / 403 errors seen when running CLI commands or in log files

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

To clarify the meaning of these errors and provide info on how and when to resolve.

Symptoms:

• Users see 403 errors mentioned in the log files or on the command line of the Salt master when executing salt commands. See example. 

  # salt-run jobs.lookup_jid 20231002141401713605 [ERROR ] Failed to get JID: 403 Forbidden Passed invalid arguments: 'NoneType' object is not iterable

• salt-master log shows an 401 error like these:
  

  .. [ERROR   ] Failed to authenticate: Authentication failed: no Authorization header
  .. [ERROR   ] Failed to save events: 401 Authentication failed: no Authorization header
  .. [ERROR   ] keyauth engine: encryption failed: RSA key format is not supported

• the public key file /etc/salt/pki/master/sseapi_key.pub is empty

 

Environment

VMware Aria Automation Config 8.x

Cause

Some errors seen in the logs are normal, but may indicate that your Salt master key (sseapi_key.pub) is expired or that the Salt master needs to be re-authenticated to Aria Config (RaaS daemon).

Resolution

Users will need to re-authenticate the Salt master to Aria Config in order to correct the issue. 

1. Stop the Salt master daemon
   1. The command "systemctl stop salt-master" will work on most systems
2. Delete the existing sseapi_key.pub file
   1. Usually found in /etc/salt/pki/master/sseeapi_key.pub
3. Login to your Aria Config UI and navigate to Administration -> Master keys
   1. NOTE: This will require a super user role to administer Salt master keys
4. Select the check box next to the Salt master key and click the "Delete Key" button near the top of the page.
5. On the CLI of the Salt master server, restart the Salt master daemon
   1. The command "systemctl start salt-master" will work on most systems
6. Navigate back to Administration -> Master keys in the UI and find the master key in the "Pending" section
7. Accept the new key
8. Confirm that you have a new public key at /etc/salt/pki/master/sseapi_key.pub
   1. Run "ls -alh /etc/salt/pki/master/sseapi_key.pub" and confirm the time stamp is new.

Additional Information

Impact/Risks:

Jobs will be queued in the UI, but will not be picked up by Salt masters and executed until the Salt masters are able to authenticate to Aria Config (RaaS daemon).

On the CLI, 403 errors will be reported for commands that reach out to Aria Config for information.