ZDU to 21.0.5 forces traces, aborts WPs, agents/users won't reconnect, finalize does not work

Products

• CA Automic Workload Automation - Automation Engine
• CA Automic One Automation

Issue/Introduction

When using the ZDU to upgrade to 21.0.5 there are numerous problems that occur:

1) after starting of new process - FORCED TRACES are written

20230214/133105.799 -  U00003631 Dump caused by: 
20230214/133105.799 -  U00003422 WARNING: Cannot load conversation memory '0e42c1a8' for server routine 'JPEXEC_R'.
20230214/133105.799 -  U00011801 Error in Server routine 'JPEXEC_R', Server: 'UC4#WP011' AE system: 'UC4'.

2) new WP aborted

20230214/131520.631 - U00022004 Starting process 'zdu.UC4 WPx' ...
20230214/131520.631 - U00022005 ... with command line  '/home/UC4/zdu-ae/bin/ucsrvwp -I/home/UC4/zdu-ae/bin/zucsrv.ini -svc(nil)ort'.
20230214/131520.631 - U00022006 ... and start directory  '/home/UC4/zdu-ae/bin'.
20230214/131520.632 - U00022007 Process 'zdu.UC4 WPx' (ID '10794') successfully started.
...
20230214/133202.005 - U00022012 Process 'zdu.UC4 WPx/UC4 PWP-Server [UC4#WP011] - 20 Connections' (ID '10794') ended.
20230214/133202.005 - U00022022 Process 'zdu.UC4 WPx/UC4 PWP-Server [UC4#WP011] - 20 Connections' ended, exit code='0(6)'.
20230214/133202.005 - U00022038 Service  'zdu.UC4 WPx' automatically starts in '60' seconds.

3) during ZDU wizard - agent and user re-connect not working

4) FINALIZE not working - throws error: There are still messages waiting in 'mq2pwp'

5)Occasionnally 21.0.5 Work processes can crash with a Segmentation fault Signal (SIGSEV 11 during processing of Events).

20230227/130353.021 - INSERT INTO MQMEM (MQMEM_PK, MQMEM_System, MQMEM_Title, MQMEM_Len, MQMEM_Content, MQMEM_Version, MQMEM_MQSet) VALUES (?, ?, ?, ?, ?, ?, ?)
20230227/130353.022 - UCUDB32 INSR RET 0000 HSTMT: 0x00000001feaf30 VALUE: 0x00000000000001 ALL:  0.00264 DB:  0.00236 ODBC:  0.00002 UDB:  0.00027
20230227/130353.022 - SendTo - UCUHOST: 0x42d600
20230227/130353.022 -   STRT UCUHOST        OPC: 0010  ucuhost-name: SCW000100712 vers=2
20230227/130353.022 -   EXIT UCUHOST        RET: 0000000000 TIME: 0000,00002 RETTEXT='' 
20230227/130353.022 - U00003409 'UC_MQ' memory successfully allocated (address = '0x7f5226d0e010', length = '538980352/526348k').
20230227/130353.023 - ActionSend(msgsize=327, SOCKET(s=16,name=UC4_UC4#WP077,type=04,host=,add=10.92.40.212,port=22100,id=0,netarea=UC4_EXP2,roles=,nxt=0x1ba4570)) -->
20230227/130353.023 - U00009909 TRACE: (Send to Server UC4_UC4#WP077)                                         0x7ffd7a1c3570 000327
                                00000000  30303030 30333237 5543343A 676C6F62  >00000327UC4:glob<
                                00000010  616C3030 314E4154 20202020 20202020  >al001NAT        <
                                00000020  20202020 20202020 20202020 20202020  >                <
                                00000030  F7140155 A87A004D 51325750 00000000  >÷..U¨z.MQ2WP....<
                                00000040  00000000 00000053 69676E61 6C203131  >.......Signal 11<
                                00000050  20285345 47562920 61742028 6E696C29  > (SEGV) at (nil)<
                                00000060  20776974 6820636F 64652031 00000000  > with code 1....<
                                00000070= 00000000 00000000 00000000 00000000  >................<
                                00000140  00000000 000000                      >.......<
20230227/130353.023 - ActionSend <-- (OK)

Environment

Release : ZDU upgrading to 21.0.5

Component: Automation Engine

Function: ZDU (Zero Downtime Upgrade)

Database Type: All Databases

OS: All OS Types

Resolution

This is fixed with 21.0.5 HF2 and higher

It is important to not use ZDU to upgrade to any lower release of 21.0 than 21.0.5 HF2.  Manual upgrades can still be carried out without issue.

Additional Information

All OS and Databases are impacted 

https://bsg-jira.broadcom.net/browse/AE-31441

https://bsg-jira.broadcom.net/browse/AE-31416

https://bsg-jira.broadcom.net/browse/AE-31508