ABX action in Aria Automation fails with timeout error

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Virtual machine deployment completes successfully; however, the associated ABX action fails with a timeout error:
  
  Action run failed with the following error: Action did not complete within 900 seconds timeout.
  
  
• The ABX action script utilizes the PowerCLI command Invoke-VMScript. Reviewing the script execution logs reveals the following error:
  
  Invoke-VMScript  The proxy tunnel request to proxy 'http://proxy-service.prelude.svc.cluster.local:3128/' failed with status code '503'."
  
  
• Additionally, the 503 error is logged in the Aria Automation proxy service at /services-logs/prelude/proxy-service/console-logs/squid-proxy.log:
  
  TCP_TUNNEL/503 0 CONNECT <ESXi ip address>:443 - HIER_DIRECT/<ESXi ip address> -

Environment

Aria Automation 8.18.x

Cause

While standard Aria Automation deployment tasks do not always require a direct connection to the underlying ESXi hosts, the Invoke-VMScript PowerCLI command functions by communicating directly with the ESXi host management IP via TCP port 443.

In this scenario, the internal proxy service (Squid) is unable to establish a tunnel to the ESXi host, leading to the 503 Service Unavailable error and the subsequent ABX timeout.

Resolution

To resolve this issue, ensure that the network firewall allows traffic between the Aria Automation appliance(s) and the ESXi host management interface.

Once the network path is unblocked, the proxy will be able to establish the necessary tunnel for the Invoke-VMScript command to execute successfully.

Additional Information

The connection from Aria Automation to ESXi host can be checked with below commands:

curl -v --noproxy \* https://<ESXi-host-ip>
kubectl exec -n prelude -it proxy-service-<press TAB to pick any pod> -- /bin/bash
curl --proxy localhost:3128 -v https://<ESXi-host-ip> 
curl -v --noproxy \* https://<ESXi-host-ip>