ZDU from version inferior to 21.0.5HF2 forces traces, aborts WPs, agents/users won't reconnect, finalize does not work

Products

• CA Automic Workload Automation - Automation Engine
• CA Automic One Automation

Issue/Introduction

When using the ZDU to upgrade to 21.0.5 there are numerous problems that occur:

1) after starting of new process - FORCED TRACES are written

20230214/133105.799 -  U00003631 Dump caused by: 
20230214/133105.799 -  U00003422 WARNING: Cannot load conversation memory '0e42c1a8' for server routine 'JPEXEC_R'.
20230214/133105.799 -  U00011801 Error in Server routine 'JPEXEC_R', Server: 'UC4#WP011' AE system: 'UC4'.

2) new WP aborted

20230214/131520.631 - U00022004 Starting process 'zdu.UC4 WPx' ...
20230214/131520.631 - U00022005 ... with command line  '/home/UC4/zdu-ae/bin/ucsrvwp -I/home/UC4/zdu-ae/bin/zucsrv.ini -svc(nil)ort'.
20230214/131520.631 - U00022006 ... and start directory  '/home/UC4/zdu-ae/bin'.
20230214/131520.632 - U00022007 Process 'zdu.UC4 WPx' (ID '10794') successfully started.
...
20230214/133202.005 - U00022012 Process 'zdu.UC4 WPx/UC4 PWP-Server [UC4#WP011] - 20 Connections' (ID '10794') ended.
20230214/133202.005 - U00022022 Process 'zdu.UC4 WPx/UC4 PWP-Server [UC4#WP011] - 20 Connections' ended, exit code='0(6)'.
20230214/133202.005 - U00022038 Service  'zdu.UC4 WPx' automatically starts in '60' seconds.

3) during ZDU wizard - agent and user re-connect not working

4) FINALIZE not working - throws error: There are still messages waiting in 'mq2pwp'

5)Occasionnally 21.0.5 Work processes can crash with a Segmentation fault Signal (SIGSEV 11 during processing of Events).

20230227/130353.021 - INSERT INTO MQMEM (MQMEM_PK, MQMEM_System, MQMEM_Title, MQMEM_Len, MQMEM_Content, MQMEM_Version, MQMEM_MQSet) VALUES (?, ?, ?, ?, ?, ?, ?)
20230227/130353.022 - UCUDB32 INSR RET 0000 HSTMT: 0x00000001feaf30 VALUE: 0x00000000000001 ALL:  0.00264 DB:  0.00236 ODBC:  0.00002 UDB:  0.00027
20230227/130353.022 - SendTo - UCUHOST: 0x42d600
20230227/130353.022 -   STRT UCUHOST        OPC: 0010  ucuhost-name: SCW000100712 vers=2
20230227/130353.022 -   EXIT UCUHOST        RET: 0000000000 TIME: 0000,00002 RETTEXT='' 
20230227/130353.022 - U00003409 'UC_MQ' memory successfully allocated (address = '0x7f5226d0e010', length = '538980352/526348k').
20230227/130353.023 - ActionSend(msgsize=327, SOCKET(s=16,name=UC4_UC4#WP077,type=04,host=,add=<ip address>,port=22100,id=0,netarea=UC4_EXP2,roles=,nxt=0x1ba4570)) -->
20230227/130353.023 - U00009909 TRACE: (Send to Server UC4_UC4#WP077)                                         0x7ffd7a1c3570 000327
                                00000000  30303030 30333237 5543343A 676C6F62  >00000327UC4:glob<
                                00000010  616C3030 314E4154 20202020 20202020  >al001NAT        <
                                00000020  20202020 20202020 20202020 20202020  >                <
                                00000030  F7140155 A87A004D 51325750 00000000  >÷..U¨z.MQ2WP....<
                                00000040  00000000 00000053 69676E61 6C203131  >.......Signal 11<
                                00000050  20285345 47562920 61742028 6E696C29  > (SEGV) at (nil)<
                                00000060  20776974 6820636F 64652031 00000000  > with code 1....<
                                00000070= 00000000 00000000 00000000 00000000  >................<
                                00000140  00000000 000000                      >.......<
20230227/130353.023 - ActionSend <-- (OK)

Environment

Release : ZDU upgrading to 21.0.5

Component: Automation Engine

Function: ZDU (Zero Downtime Upgrade)

Database Type: All Databases

OS: All OS Types

Resolution

This is fixed with 21.0.5 HF2 and higher

It is important to not use ZDU to upgrade from any lower release than 21.0.5 HF2.  Manual upgrades can still be carried out without issue.

Additionally, in case of issues during the ZDU please reach out to Technical Support and NEVER perform a manual stop of the old processes, neither a Cold-Start as that will break the ZDU process and will generate this kind of errors when trying to start the WPs:

New version:
U00015000 WP 'AUTOMIC#WP002' cannot be primary, because it is using another MQSet than the old primary WP ('1' instead of '2'). Please start a WP with an older version.
Old version:
U00011899 WP 'AUTOMIC#WP002' cannot be primary, because it is using another MQSet than the old primary WP ('2' instead of ''). Please start a WP with a newer version.

Additional Information

All OS and Databases are impacted, different Bugs were fixed around this topic:
AE-31441
AE-31416
AE-31508

In case of ZDU stuck and no processes can start correctly, this is the general procedure to restore the service:

1. On Client 0 change in UC_SYSTEM_SETTINGS: ZERO_DOWNTIME_UPGRADE from Y to N --> this usually needs to be done by a SQL statement that Technical Support will provide
2. Shutdown the AE processes from both versions
3. Launch DBLoad  with the new version uc_upd.txt
4. Start the new version WP in Cold-Start