Data collection fails with: Event Queue operation failed with MessageQueueErrorCode QueueNotFound for queue 'UpdateMissingFlag'

Products

• VMware Aria Suite

Issue/Introduction

Provides instructions to flush the dbo.WorkItem and dbo.DataCollectionStatus tables if this error is encountered.

Symptoms:

• Power On and Power Off Day 2 operations against vRealize Automation (vRA) managed machines / deployments do not process.
• Infrastructure > Monitoring > Logs detail the following similar error messages:

DC: <UUID_1>: state: vAgent: VC-Agent: Error processing data collection response, StatusID = <UUID_1>, exception: Event Queue operation failed with MessageQueueErrorCode QueueNotFound for queue 'UpdateMissingFlag'.

Environment

VMware vRealize Automation 7.x

Resolution

A resolution for this issue is not yet available.  Subscribe to this Knowledge article for updates regarding a fix in a later release. For reducing or eliminating the impact of this problem, please see the workaround below.

Workaround:
The below instructions modify the vRealize Automation IaaS SQL server database.  Ensure there is a full database backup before continuing:

Flushing the dbo.workitems table:

1. Stop all Proxy Agent services (services.msc).
2. Stop Manager services, Standby, then Primary (services.msc).
3. Using Microsoft SQL Server Management Studio (SSMS.exe), connect to the vRA IaaS SQL server database.
4. Select "New Query" and execute the following update statement:

Delete from dbo.WorkItem

5. Start Manager Services, Standby, then Primary (services.msc).
6. Start all Proxy Agents (services.msc).

Data Collections are not processing within the vRA tenant UI:

1. Stop all Proxy Agent services (services.msc).
2. Stop Manager services, Standby, then Primary (services.msc).
3. Using Microsoft SQL Server Management Studio (SSMS.exe), connect to the vRA IaaS SQL server database.
4. Select "New Query" and execute the following update statement:

Delete from dbo.DataCollectionStatus

5. Start Manager Services, Primary, then Standby (services.msc).
6. Start all Proxy Agents (services.msc).

This will force the Manager Service to regenerate the scheduling of all Data Collections. They will start triggering immediately after the services come online.

Clear out the Master Workflow Instance for all machines:

1. Stop Manager services, Standby, then Primary (services.msc).
2. Using Microsoft SQL Server Management Studio (SSMS.exe), connect to the vRA IaaS SQL server database.
3. Select "New Query" and execute the following update statement:

Delete from dbo.InstanceState

4. Start Manager Services, Primary, then Standby (services.msc).

Cleanup work items generated against managed machines persisting within the vRA tenant UI:

1. Stop Manager services, Standby, then Primary (services.msc).
2. Perform for each machine stuck in progress within the UI
   1. Confirm machine UUIID:

Select * from dbo.VirtualMachine where VirtualMachineName = '<VirtualMachineName>'

2. Execute Delete statement after confirming ID:

Delete from InstanceState where InstanceState.uidInstanceID = '<VirtualMachineID>'

3. Start Manager Services, Primary, then Standby (services.msc).