EEM is down error in WCC after JRE upgrade

Products

Autosys Workload Automation

Issue/Introduction

JRE was upgraded from out of the box version in 12.0.1 release to Java 1.8.0_341.

WCC starts up fine, but no one can login to WCC. The error users get is "EEM is down"

CA-wcc.log has an error:

INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | @tomcat-resource < status-check-1> [] ERROR #EmbIAMAccessProvider # EmbIAMAccessProvider - SafeContextFactory.getSafeContext failed for config: AccessConfig[HostName=EEM-Hostname.example.com, Locale=en_US, ServerEnabled=true, AppName=WCC0004, AppCertPath=/opt/CA/WorkloadAutomationAE/wcc/data/config/wcc.pem, ServerAdminID=EiamAdmin, EventLogPath=null, PersistentCachePath=null, RetryConnectInterval=30, RetryPingInterval=30, FullCacheUpdateEnabled=false, CacheUpdateInterval=30], SafeException.getMessage = EE_SPONSORERROR iSponsor Error
INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | [Authenticate Error: Authentication Failed]
INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | com.ca.eiam.SafeException: EE_SPONSORERROR iSponsor Error
INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | at com.ca.eiam.SafeContext.authenticateWithCertificate(SafeContext.java:1936)
INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | at com.ca.uejm.access.providers.EmbIAMAccessProvider.establishSafeContext(Unknown Source)
INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | at com.ca.uejm.access.providers.EmbIAMAccessProvider.ping(Unknown Source)
INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | at com.ca.wcc.launcher.statuscheck.EEMStatusCheck.execute(Unknown Source)
INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | at com.ca.wcc.launcher.statuscheck.StatusTask.run(Unknown Source)
INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | at sun.reflect.GeneratedMethodAccessor581.invoke(Unknown Source)
INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | at java.lang.reflect.Method.invoke(Method.java:498)
INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:283)
INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | at org.springframework.scheduling.support.MethodInvokingRunnable.run(MethodInvokingRunnable.java:69)
INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO | jvm 1 | 2022/11/06 03:09:38 | 68291 | at java.lang.Thread.run(Thread.java:827)
INFO | jvm 1 | 2022/11/06 03:09:41 | 68294 | @rest < status-check-1> [] WARN #EEMStatusCheck # Cannot ping the EEM backend ...
INFO | jvm 1 | 2022/11/06 03:09:54 | 68307 | @rest < status-check-1> [] WARN #EEMStatusCheck # Cannot ping the EEM backend ...
INFO | jvm 1 | 2022/11/06 03:10:07 | 68320 | @rest < status-check-1> [] WARN #EEMStatusCheck # Cannot ping the EEM backend ...

Switching back to original JRE, with no other changes, makes WCC work fine. No errors connecting to EEM like above, WCC can be used normally too.

Environment

Release : 12.0

Cause

As JRE got upgraded, the Java layer on WCC server was enforcing stronger encryption for the SSL connection into EEM server. EEM server still allowed TLS v1.0, TLS v1.1 and TLS v1.2, all of them, but defaulted to 1.0. The Java client layer using which WCC was making EEM calls, they were attempting TLS v.1.2. And so the problem.

1) To identify this we enabled debug on EEM SDK layer on WCC node by making a change to /opt/CA/WorkloadAutomationAE/wcc/data/config/application/config/resources/eiam.logj4.config (backup this file so it is easier to restore to default version)

<logger name="SDK" additivity="false">
<level value="trace"/>
<appender-ref ref="SDK" />
</logger>

<Root level="TRACE">
<AppenderRef ref="SDK"/>

</Root>

2) Save the file, no need to restart anything

3) Default file to monitor: /opt/CA/WorkloadAutomationAE/wcc/log/eem/eiam.sdk.java.log

4) Below error showed up in it:

DEBUG 2022-11-17 07:12:00,340 [status-check-1] [com.ca.eiam.SafeContextFactory] getSafeContext - creating new SafeContext
TRACE 2022-11-17 07:12:00,342 [status-check-1] [com.ca.eiam.SafeCache] SafeCache - Enter
TRACE 2022-11-17 07:12:00,342 [status-check-1] [com.ca.eiam.SafeCache] SafeCache - Leave
TRACE 2022-11-17 07:12:00,343 [status-check-1] [com.ca.eiam.poz.PozFactory] PozFactory - Enter
TRACE 2022-11-17 07:12:00,343 [status-check-1] [com.ca.eiam.poz.PozFactory] PozFactory - Leave
TRACE 2022-11-17 07:12:00,343 [status-check-1] [com.ca.eiam.SafeContext] SafeContext - Enter
TRACE 2022-11-17 07:12:00,344 [status-check-1] [com.ca.eiam.SafeContext] SafeContext - Exit
TRACE 2022-11-17 07:12:00,344 [status-check-1] [com.ca.eiam.SafeContext] setBackend - Enter : params[Backend=EEM-Server.example.com]
TRACE 2022-11-17 07:12:00,344 [status-check-1] [com.ca.eiam.poz.PozFactory] clearPozNodes - Enter
TRACE 2022-11-17 07:12:00,344 [status-check-1] [com.ca.eiam.poz.PozFactory] clearPozNodes - Leave
TRACE 2022-11-17 07:12:00,344 [status-check-1] [com.ca.eiam.poz.PozFactory] addPozNode - Enter : params [poz_node=EEM-Server.example.com]
DEBUG 2022-11-17 07:12:00,344 [status-check-1] [com.ca.eiam.poz.PozFactory] addPozNode - poz_node=EEM-Server.example.com, poz_nodes=[EEM-Server.example.com]
TRACE 2022-11-17 07:12:00,344 [status-check-1] [com.ca.eiam.poz.PozFactory] addPozNode - Leave
TRACE 2022-11-17 07:12:00,344 [status-check-1] [com.ca.eiam.SafeContext] SafeContext.setBackend: setting backend to EEM-Server.example.com
TRACE 2022-11-17 07:12:00,344 [status-check-1] [com.ca.eiam.SafeContext] setBackend - Getting public key runMethod
TRACE 2022-11-17 07:12:00,346 [status-check-1] [com.ca.eiam.poz.PozFactory] runMethod - Enter : params[Iclient icl,Host=EEM-Server.example.com,Sponsor=iAuthority,Method=GetPublicKey,List argList]
TRACE 2022-11-17 07:12:00,346 [status-check-1] [Network] RunBatMethod - Enter : params[Host=EEM-Server.example.com,Method=GetPublicKey,List arglist]
ERROR 2022-11-17 07:12:00,397 [status-check-1] [Network] RunBatMethod - exception occurred calling Iclient runBatMethod
com.ca.itechnology.iclient.IclException: javax.net.ssl.SSLHandshakeException: The server selected protocol version TLS10 is not accepted by client preferences [TLS12]
at com.ca.itechnology.iclient.IclUtil.httpRequest(IclUtil.java:1544)
at com.ca.itechnology.iclient.IclMethod.runBatMethod(IclMethod.java:212)
at com.ca.itechnology.iclient.Iclient.runBatMethod(Iclient.java:580)
at com.ca.eiam.poz.Network.RunBatMethod(Network.java:320)
at com.ca.eiam.poz.PozFactory.runMethod(PozFactory.java:2236)
at com.ca.eiam.SafeContext.setBackend(SafeContext.java:5707)
at com.ca.eiam.SafeContextFactory.createSafeContext(SafeContextFactory.java:173)
at com.ca.eiam.SafeContextFactory.getSafeContext(SafeContextFactory.java:123)
at com.ca.uejm.access.providers.EmbIAMAccessProvider.establishSafeContext(Unknown Source)
at com.ca.uejm.access.providers.EmbIAMAccessProvider.ping(Unknown Source)
at com.ca.wcc.launcher.statuscheck.EEMStatusCheck.execute(Unknown Source)
at com.ca.wcc.launcher.statuscheck.StatusTask.run(Unknown Source)
at sun.reflect.GeneratedMethodAccessor581.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:283)
at org.springframework.scheduling.support.MethodInvokingRunnable.run(MethodInvokingRunnable.java:69)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:827)
Caused by: javax.net.ssl.SSLHandshakeException: The server selected protocol version TLS10 is not accepted by client preferences [TLS12]
at sun.security.ssl.Alert.createSSLException(Alert.java:131)
at sun.security.ssl.Alert.createSSLException(Alert.java:117)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:311)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:267)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:258)
at sun.security.ssl.ServerHello$ServerHelloConsumer.onServerHello(ServerHello.java:943)
at sun.security.ssl.ServerHello$ServerHelloConsumer.consume(ServerHello.java:869)
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377)
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:152)
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1397)
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1305)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:818)
at sun.security.ssl.SSLSocketImpl.access$200(SSLSocketImpl.java:73)
at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1180)
at org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:160)
at org.apache.http.impl.io.AbstractSessionOutputBuffer.flush(AbstractSessionOutputBuffer.java:168)
at org.apache.http.impl.io.ContentLengthOutputStream.flush(ContentLengthOutputStream.java:99)
at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:115)
at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:94)
at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:110)
at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:118)
at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:266)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:214)
at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:679)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:481)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at com.ca.itechnology.iclient.IclUtil.httpRequest(IclUtil.java:1509)
... 24 more

Resolution

Enable TLS 1.2 on iGateway

1) Backup the /opt/CA/SharedComponents/iTechnology/igateway.conf file

2) Edit original /opt/CA/SharedComponents/iTechnology/igateway.conf

3) Search for string <secureProtocol/> in that file

4) Modify it to look like : <secureProtocol>TLSv1_2</secureProtocol>

5) Save the file and restart igateway

./S99igateway stop ; sleep 1 ; ./S99igateway start

netstat -an |grep 5250 ## to verify the port is now in LISTEN mode

8) Verify EEM URL can now be accessed via a browser now

9) Then verify that the WCC login works fine now, the errors should now disappear

10) Restore original /opt/CA/WorkloadAutomationAE/wcc/data/config/application/config/resources/eiam.logj4.config file to disable debug.