Minion grain values are not updating or show incorrect value in Aria Config

Products

• VMware Tanzu Platform Services - SALT
• VCF Operations/Automation (formerly VMware Aria Suite)
• VCF Automation

Issue/Introduction

Grains in the Aria Config UI (RaaS) are not showing up, or show an incorrect value

• User has HA masters and the output from salt-run cache.grains '*' from each master does not match
  • This means two or more Salt masters that minions are configured to connect to simultaneously
• User is using a custom grain configured either in the minion configuration or set via a custom grains module.
• When users run salt <MINION_ID> grains.item <SPECIFIC_GRAIN_KEY> they get the expected value as the result

Environment

Aria Config 8.14+

Salt 3006.x +

Cause

The root cause for this might be one of the following issues

• The minion is not connecting to both masters consistently
  • This might be due to one or both masters becoming overloaded
• User is using a custom grain value that initially starts with a default value like 'Unknown'
• User runs a health check script that potentially restarts the Salt minion too often

Resolution

• Confirm that health check scripts are not running any commands like salt-call saltutil.sync_all
  • salt-call commands, although initiated from the Salt minion, communicate back to the Salt master. So having a cron job set in your infrastructure configured at the same time across all minions will essentially DOS your Salt master. This is known as the "Thundering herd" problem.
  • Ensure that the minion is configured appropriately with startup/reconnect settings
    • See Salt Minion Connection Settings for more info
• Ensure that the minion is able to communicate with both masters
  • Run salt <MINION_ID> test.ping --summary from each Salt master
  • The result should appear in the UI for both commands