After applying the CUM 3 Patch on windows 2019 none of the AE service will start.
search cancel

After applying the CUM 3 Patch on windows 2019 none of the AE service will start.

book

Article ID: 253703

calendar_today

Updated On:

Products

Autosys Workload Automation CA Workload Automation AE - Scheduler (AutoSys)

Issue/Introduction

After applying the CUM3 Patch to our R12 SP1 installation none of the services will start.
We are seeing the following in the System Application Event logs

 

Log Name:      Application
Source:        Application Error
Date:          10/30/2022 11:47:29 AM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      astgensys1
Description:
Faulting application name: as_server.exe, version: 12.0.1.78, time stamp: 0x62c9c3c7
Faulting module name: ntdll.dll, version: 10.0.17763.3232, time stamp: 0xd6e0b8e1
Exception code: 0xc0000005
Fault offset: 0x000000000005e4eb
Faulting process id: 0x29d0
Faulting application start time: 0x01d8ec900f0e4b8b
Faulting application path: C:\Program Files\CA\WorkloadAutomationAE\autosys\bin\as_server.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: ffdf8d7e-8b6b-4638-be95-40ca8e7e083a
Faulting package full name: 
Faulting package-relative application ID: 
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2022-10-30T18:47:29.547141500Z" />
    <EventRecordID>70106</EventRecordID>
    <Channel>Application</Channel>
    <Computer>astgensys1.company.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>as_server.exe</Data>
    <Data>12.0.1.78</Data>
    <Data>62c9c3c7</Data>
    <Data>ntdll.dll</Data>
    <Data>10.0.17763.3232</Data>
    <Data>d6e0b8e1</Data>
    <Data>c0000005</Data>
    <Data>000000000005e4eb</Data>
    <Data>29d0</Data>
    <Data>01d8ec900f0e4b8b</Data>
    <Data>C:\Program Files\CA\WorkloadAutomationAE\autosys\bin\as_server.exe</Data>
    <Data>C:\Windows\SYSTEM32\ntdll.dll</Data>
    <Data>ffdf8d7e-8b6b-4638-be95-40ca8e7e083a</Data>
    <Data>
    </Data>
    <Data>
    </Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Application Error
Date:          10/30/2022 11:47:34 AM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      astgensys
Description:
Faulting application name: event_demon.exe, version: 12.0.1.78, time stamp: 0x62c9c3c5
Faulting module name: ntdll.dll, version: 10.0.17763.3232, time stamp: 0xd6e0b8e1
Exception code: 0xc0000005
Fault offset: 0x000000000005e4eb
Faulting process id: 0x2c4
Faulting application start time: 0x01d8ec90129e46f9
Faulting application path: C:\Program Files\CA\WorkloadAutomationAE\autosys\bin\event_demon.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 30158a5c-58a5-4147-9156-7bfc156db5c3
Faulting package full name: 
Faulting package-relative application ID: 
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2022-10-30T18:47:34.212263500Z" />
    <EventRecordID>70109</EventRecordID>
    <Channel>Application</Channel>
    <Computer>astgensys1.company.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>event_demon.exe</Data>
    <Data>12.0.1.78</Data>
    <Data>62c9c3c5</Data>
    <Data>ntdll.dll</Data>
    <Data>10.0.17763.3232</Data>
    <Data>d6e0b8e1</Data>
    <Data>c0000005</Data>
    <Data>000000000005e4eb</Data>
    <Data>2c4</Data>
    <Data>01d8ec90129e46f9</Data>
    <Data>C:\Program Files\CA\WorkloadAutomationAE\autosys\bin\event_demon.exe</Data>
    <Data>C:\Windows\SYSTEM32\ntdll.dll</Data>
    <Data>30158a5c-58a5-4147-9156-7bfc156db5c3</Data>
    <Data>
    </Data>
    <Data>
    </Data>
  </EventData>
</Event>

Environment

Autosys R12 SP1 CUM3

Cause

The CAPKI5 had not been properly updated to the new version.

In the CAPKI directory, the client was missing the Microsoft.VC80.CRT directory and files

Directory of C:\Program Files\CA\SharedComponents\CAPKI\CAPKI5\Windows\amd64\64\lib

11/01/2022  08:12 AM    <DIR>          .
11/01/2022  08:12 AM    <DIR>          ..
10/31/2022  10:58 AM         1,928,648 libcaopenssl_crypto.dll
10/31/2022  10:58 AM           367,560 libcaopenssl_ssl.dll
10/31/2022  10:58 AM           873,416 libcapki.dll
10/31/2022  10:58 AM            21,960 libcapki_ipthread.dll
10/31/2022  10:58 AM            18,888 libcapki_thread.dll
10/31/2022  10:58 AM    <DIR>          Microsoft.VC80.CRT
               5 File(s)      3,210,472 bytes

 Directory of C:\Program Files\CA\SharedComponents\CAPKI\CAPKI5\Windows\amd64\64\lib\Microsoft.VC80.CRT

10/31/2022  10:58 AM    <DIR>          .
10/31/2022  10:58 AM    <DIR>          ..
10/31/2022  10:58 AM             1,871 Microsoft.VC80.CRT.manifest
10/31/2022  10:58 AM           516,096 msvcm80.dll
10/31/2022  10:58 AM         1,061,376 msvcp80.dll
10/31/2022  10:58 AM           796,672 msvcr80.dll
               4 File(s)      2,376,015 bytes

Checking the registry version for the CAPKI5 the client showed the following:

[HKEY_LOCAL_MACHINE\SOFTWARE\ComputerAssociates\Shared\CAPKI5]
"InstallDir"="C:\\Program Files\\CA\\SharedComponents\\CAPKI"
"Version"="5.2.6"

[HKEY_LOCAL_MACHINE\SOFTWARE\ComputerAssociates\Shared\CAPKI5\Dependencies]
"WAAE"="5.2.6"
"ETRDIR"="5.2.4"
"iGateway"="5.2.2"

 

This should have been upgraded to the following:

[HKEY_LOCAL_MACHINE\SOFTWARE\ComputerAssociates\Shared\CAPKI5]
"InstallDir"="C:\\Program Files\\CA\\SharedComponents\\CAPKI"
"Version"="5.2.9"

[HKEY_LOCAL_MACHINE\SOFTWARE\ComputerAssociates\Shared\CAPKI5\Dependencies]
"WAAE"="5.2.9"
"ETRDIR"="5.2.4"
"iGateway"="5.2.2"

CAPIK5 was determined to be the issue by reviewing the CORE dump from the application startup.

The dump file shows that the event_demon is crashing while loading CAPKI module at startup:

 

00 000000e0`6093c538 00007ffb`1c09d43e     ntdll!ZwWaitForMultipleObjects+0x14
01 000000e0`6093c540 00007ffb`1c09d32e     KERNELBASE!WaitForMultipleObjectsEx+0xfe
02 000000e0`6093c840 00007ffb`1c9f7c7f     KERNELBASE!WaitForMultipleObjects+0xe
03 000000e0`6093c880 00007ffb`1c9f772c     kernel32!WerpLaunchAeDebug+0x285f
04 000000e0`6093c9a0 00007ffb`1c0c7d3e     kernel32!WerpLaunchAeDebug+0x230c
05 000000e0`6093c9e0 00007ffb`1f4f9180     KERNELBASE!UnhandledExceptionFilter+0x35e
06 000000e0`6093caf0 00007ffb`1f4dfdd6     ntdll!memset+0x1c40
07 000000e0`6093cb30 00007ffb`1f4f575f     ntdll!_C_specific_handler+0x96
08 000000e0`6093cba0 00007ffb`1f454cef     ntdll!_chkstk+0x11f
09 000000e0`6093cbd0 00007ffb`1f4f44ce     ntdll!RtlWalkFrameChain+0x14bf
0a 000000e0`6093d300 00007ffb`1f4ae4eb     ntdll!KiUserExceptionDispatcher+0x2e
0b 000000e0`6093da20 00007ffb`1f46449c     ntdll!RtlDllShutdownInProgress+0x9b
0c 000000e0`6093da90 00007ffb`1f4643b0     ntdll!RtlEnterCriticalSection+0x12c
0d 000000e0`6093dac0 00007ffb`1b7988d6     ntdll!RtlEnterCriticalSection+0x40
0e 000000e0`6093daf0 00007ffb`1b798984     ucrtbase!_stdio_common_vsscanf+0x3f6
0f 000000e0`6093db20 00007ffb`13001dad     ucrtbase!_stdio_common_vfprintf+0x74
10 000000e0`6093db90 00007ffb`12fa1935     libascmn!AsString::xtright+0x9d
11 000000e0`6093dbe0 00007ffb`12fa5a41     libascmn+0x1935
12 000000e0`6093e080 00007ffb`13018627     libascmn+0x5a41
13 000000e0`6093e0b0 00007ffb`130159e6     libascmn!AutoEtpki::importKey+0x257
14 000000e0`6093e110 00007ffb`1301928f     libascmn!AutoEtpki::AutoEtpki+0xb6
15 000000e0`6093e1c0 00007ffb`11fb8687     libascmn!AutoPass::AutoPass+0x2f
16 000000e0`6093e1f0 00007ffb`11fa9e20     libcxmgr!ConnectionMgr::getUserPw+0x487
17 000000e0`6093ec90 00007ffb`11fcf49a     libcxmgr!ConnectionMgr::connect+0x190
18 000000e0`6093f240 00007ff7`5978ff82     libcxmgr!CmPool::startup+0x1ea
19 000000e0`6093f360 00007ff7`59794b44     event_demon!AsSingleton<AsLibCurlWrapper>::GetSingleton+0x8b872
1a 000000e0`6093f8c0 00007ffb`1c9a7974     event_demon!AsSingleton<AsLibCurlWrapper>::GetSingleton+0x90434
1b 000000e0`6093f900 00007ffb`1f4aa2f1     kernel32!BaseThreadInitThunk+0x14
1c 000000e0`6093f930 00000000`00000000     ntdll!RtlUserThreadStart+0x21

Resolution

It appears that the CAPKI component was not installed correctly when the patch was installed.  
The version of CAPKI for WAAE is incorrect and the binaries under CAPKI dir are from 2021.

The patch will have a folder 'capki' under which there is setup.exe that needs to be run as below from an administrative command prompt:

setup.exe install caller=WAAE instdir="%CASHCOMP%" verbose env=all

Run this under autosys command prompt and ensure that %CASHCOMP% is set correctly to 'C:\Program Files\CA\SharedComponents\'

 

Ensure that the binaries are updated and the registry is also updated with WAAE version correctly as 5.2.9.  
It is currently at 5.2.6.

Once CAPKI is upgraded correctly, start the AE services.

Additional Information

if dump file creation is not enabled, here is the procedure to enable dump files:

 

  1. Create a new key in the registry as event_demon.exe 

Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\event_demon.exe 

  1. Under the event_demon.exe key create four new variables as 

Name 

Type 

Value 

DumpFlags 

DWORD 

0x1000 

DumpCount 

DWORD 

0xa  

DumpType 

DWORD 

2 

DumpFolder 

STRING 

The folder where you want the dump to be collected 
Powered by Wolken Software