Deployments fail with 30 minute timeout on Minion deployment stage.

Products

• VMware
• VMware Aria Suite

Issue/Introduction

Symptoms:

• VMware Aria Automation Config (formerly vRealize Automation SaltStack Config) deployments fail with 30-minute timeout on Minion deployment stage of build when deploying Minions onto a SSE Master Cluster. 
• When this issue occurs, everything looks OK in the Salt UI - deploy.minion job completes successfully.
  • When we check the minion, it looks good as well. However, something is failing and never reporting success back to Aria Automation Config.
• Error in the logs:

  Resource [/resources/compute/<UUID_1>]:: Salt resource with job id [<UUID_1>] failed. Error:: Timeout: Salt server did not respond in 30 minutes for job id [<UUID_1>]

• Possible Error in UI
  
  (SaltStack resource with job id [xxx] failed.  Error: : xxx xxx present in All minions group | xxx: State file run started. JID -xxx | xxx:  Maximum time allowed for minion deployment exceeded.  xxx: minion deployment and/or state failed.)
  

Environment

VMware Aria Automation Config 8.x
VMware vRealize Automation SaltStack Config 8.11.x
VMware Aria Automation Config 8.12.x

Cause

During Salt resource creation, there is a 30-minute timeout for each Salt task polls for the status of the salt job for the below:

• deploy.minion
• key.accept
• state.apply

If the status of the task in progress (ex: state.apply) has not been updated in 30 minutes, the salt resource creation will fail. This has been noted as an issue when there are long running state files.

Resolution

This issue is resolved in VMware Aria Automation Config 8.12.1.

To workaround the timeout:

• In the Cloud.SaltStack section of the Aria Automation blueprint you can specify createTimeout the can either have values that end in m (for minutes) or h (for hours). See the example below:
   

  Cloud_SaltStack:
      type: Cloud.SaltStack
   ...
      # Field to extend timeout
      createTimeout: 1h
      ...

• In the /etc/raas/raas.conf the additional parameters can be added as well. These changes will take effect once the raas service has been restarted (systemctl restart raas)

  sseapi_win_minion_deploy_delay: 360      # Sets a delay to allow all requisite Windows services to become active. The default value is 180 seconds
  sseapi_linux_minion_deploy_delay: 120    # Sets a delay to allow all requisite Linux services to become activate. The default value is 90 seconds.
  
  # Minion deployment settings
  minion_deployment:
    max_minion_deployment_time: 3600      # Maximum time (in seconds) allowed for minion deployment after which status will be marked as failed
    airgap_install: false                                  # Deploy minions in an airgapped environment

• Also see Troubleshoot minion deployments

.