All Work Processes are "Grayed out" in AWI and cannot be manually restarted or interacted with. SvcMgrDialog shows all green.

Products

• CA Automic One Automation

Issue/Introduction

Looking at our Automic Work Processes from within AWI, they are all showing "grayed out", even though they are showing "Green" & active in the Service Manager Dialog.  We did experience a Database Failover event just prior to this happening.

Jobs are running but there are no WPs showing active in AWI.  The WP Utilization is also high - 50% plus compared to 2% normally.

 

Environment

Release : 12.3

Component : AE/AWI/Service Manager Dialog

Cause

This was a two-node AE system

During the DB failover, which took approximately 10 minutes, the node that was thought NOT to be the node with the PWP was restarted. 

Because the DB was down and the node that was the PWP was restarted, the ownership of the PWP process could not switch appropriately.

After the database came back up, it showed the PWP was on node 2, however, the port was 0 and not 2271 as it should have been.

The query -- select * from mqsrv where mqsrv_type = 4

will show what server the PWP is running on, and as well will show the port that is being used. If the port is "0" then the switch did not happen correctly. 

Resolution

A full restart (normal) of both nodes resolved the issue. This was done with a delay to the start of node 2. This delay included a verification that node 1 correctly took ownership of the PWP process and that AWI now showed all WPs as expected. Once node 1 was verified, node 2 was started and everything was functioning as expected. 

Additional Information

In this instance, the PWP was thought to be running on node 2, and node 1 was restarted in error.