Within the vRealize Automation Events tab, repetitious error messages continue to spam: "Request to vCO failed. Error : 404"

Products

• VMware Aria Suite

Issue/Introduction

• The Event tab contains the following repetitious spamming error messages:

Request to vCO failed. Error : 404

• Access log entries, /var/log/vmware/vco/app-server/localhost_access_log.txt, similar to:

2017-12-06 12:00:12.220+0100 [https-jsse-nio-0.0.0.0-8281-exec-8] 10.164.13.247 - - "GET /vco/api/workflows/<Workflow_ID>/executions/<Workflow_Execution_ID>/ HTTP/1.1" 404 167 6
2017-12-06 12:00:12.221+0100 [https-jsse-nio-0.0.0.0-8281-exec-2] 10.164.13.247 - - "GET /vco/api/workflows/<Workflow_ID>/executions/<Workflow_Execution_ID>/ HTTP/1.1" 404 167 6
2017-12-06 12:00:12.222+0100 [https-jsse-nio-0.0.0.0-8281-exec-7] 10.164.13.247 - - "GET /vco/api/workflows/<Workflow_ID>/executions/<Workflow_Execution_ID>/ HTTP/1.1" 404 167 9
2017-12-06 12:00:12.225+0100 [https-jsse-nio-0.0.0.0-8281-exec-13] 10.164.13.247 - - "GET /vco/api/workflows/<Workflow_ID>/executions/<Workflow_Execution_ID>/ HTTP/1.1" 404 167 6
2017-12-06 12:00:12.226+0100 [https-jsse-nio-0.0.0.0-8281-exec-3] 10.164.13.247 - - "GET /vco/api/workflows/<Workflow_ID>/executions/<Workflow_Execution_ID>/ HTTP/1.1" 404 167 6
2017-12-06 12:00:12.226+0100 [https-jsse-nio-0.0.0.0-8281-exec-17] 10.164.13.247 - - "GET /vco/api/workflows/<Workflow_ID>/executions/<Workflow_Execution_ID>/ HTTP/1.1" 404 167 6

Environment

VMware vRealize Automation 7.x

Resolution

Ensure there are valid backups of the vRealize Automation database as the below instructions edit the database.  In memory snapshots are not an efficient backup solution.  Utilize one of the following: snapshot from a fully shutdown state or 3rd party solutions that may backup the VMDK.

1. The IDs of stuck workflows can be retrieved via:

SELECT * FROM o11n_workflowrunstate
WHERE executionstate='running';

2. Isolate the faulty records:

SELECT * from o11n_workflowrunstate WHERE id IN ('ExecutionIDFromLogs','ExecutionIDFromLogs','ExecutionIDFromLogs');

3. Delete the offending records found from the above queries and logs:

DELETE from o11n_workflowrunstate WHERE id IN ('ExecutionIDFromLogs','ExecutionIDFromLogs','ExecutionIDFromLogs');

Additional Information

The issue is caused by polls made to unavailable workflow executions by vRealize Automation through the  o11n-gateway service to vRealize Orchestrator.