Aria Operations agent Installation is failing with Error Unable to connect to Salt Master

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

In the Agent user interface in the Aria Operations instance, the agent will show the status "Agent unhealthy" and the "Last Operation status": "Install failed" or "Content upgrade failed ".
When you hover the mouse over the error on  "Last Operation status" it shows any of the following errors

• 14/17 Failed to connect to Salt master. Verify Agent installation log for more details.
• ERROR: Minions returned with non-zero exit code
• Failed to Connect to Salt Master. Verify Agent installation log for more details.
• Bootstrap failed for VM <VM-ID> at stage 1 with error message Failed to download: https://<IP or FQDN of cloud proxy>/downloads/salt/config-utils.bat

Environment

Aria Operation 8.18.X

Resolution

NOTE: Before following the steps below ensure that a snapshot is present for the Cloud Proxy

1. Verify that Cloud proxy salt master ports (4505,4506) are reachable from the Endpoint VM and the salt master runs properly.
   
   To check If the required ports are open, use the command(s) below:
   
   For Linux Virtual machine:
   timeout 10 bash -c "</dev/tcp/<IP or FQDN of cloud proxy>/<port>"

For Windows Virtual machine:
curl -k -I -L https://<IP or FQDN of cloud proxy>:<port>

   Note: You will see the message "Could not connect to server" when using curl if the port is not open.
   
   
   
2. Check if salt master docker is running on cloud proxy, ensure all salt master processes are running as per the image below:
   
   Run this command on Cloud proxy:
docker exec -it ucp-controlplane-saltmaster bash -c "ps -ef|grep salt"
   
   
   
   
   
3. On Cloud proxy: please verify the ownership of the folders in /ucp/salt/pki/master.
   File ownership should be admin:admin as per below:
   
   drwxr-xr-x 2 admin admin 4096 May 24 06:15 minions_pre
drwxr-xr-x 2 admin admin 4096 May 24 06:15 minions_denied
drwxr-xr-x 2 admin admin 4096 May 24 06:15 minions
drwxr-xr-x 2 admin admin 4096 May 24 06:15 minions_rejected
drwxr-xr-x 2 admin admin 4096 May 24 06:15 minions_autosign
-r-------- 1 admin admin 1678 May 24 06:15 master.pem
-rw-r--r-- 1 admin admin 450 May 24 06:15 master.pub
   
   If the files are owned by root, reset the permissions by running the /ucp/ucp-config-scripts/ucp-firstboot.sh script and verify the permissions for the folder.
   
   
   
4. After completing steps 1,2 and 3 please retry the agent installation.