Salt-minion configuration is wiped and service is disabled after restarting Aria Operations for Logs

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

After rebooting the operating system of a VMware Aria Operations for Logs node, or after manually restarting the service using systemctl restart loginsight, users encounter the following issues:

• The Salt-minion service fails to start automatically.

• The configuration file /etc/salt/minion.d/master.conf is truncated to 1 byte and contains no data.

• The salt-minion service status shows as disabled and inactive (dead) despite being enabled prior to the restart.

• The /var/log/salt/minion log file contains the following entry: salt-minion[44048]: The Salt Minion is shutdown. Minion received a SIGTERM. Exited.

Environment

 

• Product: VMware Aria Operations for Logs (formerly vRealize Log Insight)

• Version: 8.18.3

• Component: Salt-minion

• Salt Version: 3006.8

 

Cause

Aria Operations for Logs version 8.18.x added the ability to configure Salt via the API. In this updated version, during a service restart or system reboot, a SIGTERM stops and disables the salt-minion service and a cleanup script targets the Salt configuration directory, truncates the master.conf file, and disables the startup of the salt-minion.

From the API documentation at https://FQDNforAriaOperationsforLogs>/rest-api/supported#/salt/POST_salt:

This API allows to configure the following four fields: master, master_type, retry_dns, and retry_dns_count.

Note that this API is only aware of these specific fields and will not recognize any additional fields in the /etc/salt/minion.d/master.conf file. Starting from version 8.18, any configuration details added directly to this file will not be recognized by the API.

 

Resolution

To resolve using the swagger API

 To configure the salt-minion through the REST API.

1. Log into the Aria Operations for Logs REST API at https://FQDNforAriaOperationsforLogs>/rest-api/supported#

2. Authorize your API session by scrolling down to /sessions/current. Select Try it out and then Execute. You will be presented with a bearer token. Copy this token to your clipboard.

3. Scroll up until you see the Authorize button:

4. Click Authorize and paste the token into this field.

5. Scroll down until you find the salt API entry. It should be at the bottom of the list of API commands.

6. Configure the salt-minion using the API Post command.

The following are example entries:

{
  "master": [
    "<salt-master-1 FQDN>",
    "<salt-master-2 FQDN>"
  ],
  "masterType": "str",
  "retryDns": 10,
  "retryDnsCount": 3
}

Note: You only need to change the <salt-master-x FQDN> values. Leave the rest of the payload as is.

7. Select Try it out and then Execute.

If successful, you should see the message: 200 HTTP response code 200 indicates that salt was configured successfully.

Once configured via API, the salt-minion service will remain running and the configuration persists across service restarts or VM reboots.

Alternate method using curl

If you wish to use curl commands:

1. Log into Aria Operations for Logs using an administrative account.

2. Open an ssh session as root into any of the cluster nodes.

3. Use the following to curl command derive your auth (bearer) token:

curl -X 'GET' \
  'https://IPorFQDNofAO4L:9543/api/v2/sessions/current' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer ########-####-####-####-############'

Note: Replace IPorFQDNofAO4L with the IP or FQDN for your Aria Operations for Logs instance.

4. Apply your new salt-minion settings by sending the following curl command:

curl -X 'POST' \
  'https://IPorFQDNofAO4L:9543/api/v2/salt' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer ########-####-####-####-############' \
  -H 'Content-Type: application/json' \
  -d '{
  "master": [
    "salt-master-1 FQDN",
    "salt-master-2 FQDN"
  ],
  "masterType": "str",
  "retryDns": 10,
  "retryDnsCount": 3
}'

Note: Replace IPorFQDNofAO4L with the IP or FQDN for your Aria Operations for Logs instance.

 

Additional Information

Other pieces of the salt-minion configuration like the minion.conf and any other grains are not affected by this API call or the start/restart/reboot functions.