Custom Aria Orchestrator action utilizing createMachineTags action fails with read timeout

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

When attempting to update or add tags to a vRA (VMware Aria Automation) Virtual Machine using the out-of-the-box vRO (Aria Orchestrator) action com.vmware.library.vra.infrastructure.machine.createMachineTags, or by invoking a PATCH request using API to https://<vRA_FQDN>/iaas/api/machines/<MachineID, the following symptoms occur:

• The vRO workflow fails with the error message: java.lang.RuntimeException: Read timed out.

• The error typically occurs after exactly 120 seconds (the default vra.plugin.operation.timeout).

• Despite the error in vRO, the tags may eventually appear on the VM in vCenter after a few minutes (e.g., 3–4 minutes).

• The issue is more prevalent in environments with high scale (e.g., clusters with >1,000 VMs or vCenters with >4,000 total tags).

Environment

VMware Aria Orchestrator 8.x

VMware Aria Automation 8.x

Cause

The issue is caused by the performance limitations of the vSphere Tagging API during lookup operations combined with how the vRA IaaS API processes requests:

1. Provisioning API : The com.vmware.library.vra.infrastructure.machine.createMachineTags action uses the provisioning API. This API treats the request as a "desired state" operation. Even if you are adding only one tag, the system may validate or look up all existing tags associated with the machine.

2. vSphere API Latency: In large environments, each tag lookup can take 15–20 seconds. If a VM has 15–20 tags, the total processing time exceeds the 120-second timeout window.

3. Timeout Chain: vRO waits for the IaaS API, which waits for the Provisioning Service, which waits for the vSphere Adapter. If the vSphere tagging process takes 3–4 minutes, the initial vRO request is abandoned due to the timeout.

Resolution

This issue is under review with Broadcom Engineering.

At this time, The recommended solution is to move away from the provisioning API (action:com.vmware.library.vra.infrastructure.machine.createMachineTags)  for tagging and instead use the vRA Deployment API.

1. Use the Deployment API (Recommended)

Unlike the provisioning API, the vRA Deployment API (/deployment/api/resources/{resourceId}/requests) is designed to send a delta (only the modified tags) for processing.

• Benefit: By sending only the new or changed tags, the number of vSphere API lookups is significantly reduced, allowing the operation to complete well within the default timeout.

• Action: Update custom workflows to call the Deployment Resource Request API instead of the infrastructure machine action.

2. Batching Large Tag Updates

If you must add a very large number of tags to a single resource, it is advised to perform these associations in small batches. This ensures that even with vSphere latency, no single request exceeds the timeout limit.

Additional Information

Related Information

• vCenter 8.0 U3 Tagging Performance: VMware Performance Documentation

 

Increase Plugin Timeout(in seconds):

• cmd:

  vracli vro properties set -k vra.plugin.operation.timeout -v 120

  Max 4 Min, any more than that would by default be timed out by Envoy Gateway (5 min Default timeout)