vCenter Server Proxy Agent in vRA stops without reporting any errors

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• In VMware vRealize Automation, the vCenter Server Proxy Agent fails to process additional work items.
• There are no errors in any logs indicating a failure.
• The Agent log shows that the Agent is responsive and pinging as expected.

Environment

VMware vRealize Automation 7.0.x
VMware vCloud Automation Center for Desktop 5.2.x
VMware vCloud Automation Center for Server 6.0.x
VMware vCloud Automation Center for Desktop 6.1.x
VMware vCloud Automation Center for Server 5.2.x
VMware vCloud Automation Center for Server 6.1.x
VMware vRealize Automation 6.2.x
VMware vRealize Automation Desktop 6.2.x
VMware vCloud Automation Center for Desktop 6.0.x

Cause

This issue occurs if there are many CustomizeVM tasks stuck in the WorkItem table and cannot be completed. When a provisioning machine is vMotioned during provisioning, it may fail to report back to the correct host causing the CustomizeVM task to fail.

To verify this, check in the Manager Service All.log file for CustomizeVM tasks that have started but have not completed. The Proxy Agent can only queue up a maximum of 11 work items. If 11 of these tasks are stuck, the Proxy Agent stops processing new tasks.

Resolution

This is a known issue affecting VMware vRealize Automation 5.2.x , 6.x and 7.x.

Currently, there is no resolution.

 

To work around this issue, remove the entries corresponding to the failed CustomizeVM tasks from the dbo.WorkItem table in the vRealize Automation SQL database.

To remove the entries corresponding to the failed CustomizeVM tasks from the dbo.WorkItem table:

Note: This procedure modifies the database. Ensure to take a backup of the database before proceeding.

1. Stop the Proxy Agent service on the IaaS server(s).
2. Back up the vRealize Automation SQL database.
3. Query the dbo.WorkItem table to identify any stuck CustomizeVM tasks.
4. Remove the failed CustomizeVM rows from the dbo.WorkItem table.
5. Start the Proxy Agent service and monitor the logs for new work items to be queued up.

Additional Information