UC_EXECUTOR_END task goes to inconsistent after upgrade of AE

Products

• CA Automic Workload Automation - Automation Engine

Issue/Introduction

After upgrading Automic manually, the task kicked off by UC_EXECUTOR_END goes to an inconsistent state.  Steps to upgrade are usually:

Stop all AE processes and agents
Upgrade components (including initialdata)
Start up all processes

Notice when logging in that the object started by the UC_EXECUTOR_END function is now in an inconsistent status in Process Monitoring.

Environment

Release : 12.3

Component :

Resolution

When reviewing the WP and CP logs, the object started using the UC_EXECUTOR_END task was not activated until after all CPs ended.  When an upgrade to the initialdata is done manually (not using ZDU), a cold start is run automatically and will put all tasks in the middle of Automic scripting into an inconsistent status.  This is by design as the message queue tables that have what should be done via that scripting is removed and the Automation Engine no longer knows what to do.

Because the UC_EXECUTOR_END tasks were in the middle of the Automic scripting when the last WP stopped, everything the system needed to know about those tasks was held in our Message Queue or MQ tables - these are tables for temporary information that drive everything in the system.  When the upgrade was done and the initialdata upgraded, the system was put through a COLD start that clears out the MQ tables when the system started up.  So when the system started up, it did not have any more instructions on what to do with those, but it also knew they had not ended, so it puts them into an inconsistent status.

To prevent this, the CPs and WPs should be stopped quite a while after stopping all agents.  This way the UC_EXECUTOR_END tasks have enough time to run and finish.