EEM is down error in WCC after JRE upgrade

Products

• CA Workload Automation AE

Issue/Introduction

Customer upgraded JRE from out of the box one in 12.0.1 release to   Java 1.8.0_341.   WCC starts up fine, but no one can login to WCC.  They get error  "EEM is down"

CA-wcc.log has an error:

 

 

INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 | @tomcat-resource < status-check-1> [] ERROR #EmbIAMAccessProvider               # EmbIAMAccessProvider - SafeContextFactory.getSafeContext failed for config: AccessConfig[HostName=EEM-Hostname.Company.com, Locale=en_US, ServerEnabled=true, AppName=WCC0004, AppCertPath=/opt/CA/WorkloadAutomationAE/wcc/data/config/wcc.pem, ServerAdminID=EiamAdmin, EventLogPath=null, PersistentCachePath=null, RetryConnectInterval=30, RetryPingInterval=30, FullCacheUpdateEnabled=false, CacheUpdateInterval=30], SafeException.getMessage = EE_SPONSORERROR iSponsor Error
INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 | [Authenticate Error: Authentication Failed]
INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 | com.ca.eiam.SafeException: EE_SPONSORERROR iSponsor Error
INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 |  at com.ca.eiam.SafeContext.authenticateWithCertificate(SafeContext.java:1936)
INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 |  at com.ca.uejm.access.providers.EmbIAMAccessProvider.establishSafeContext(Unknown Source)
INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 |  at com.ca.uejm.access.providers.EmbIAMAccessProvider.ping(Unknown Source)
INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 |  at com.ca.wcc.launcher.statuscheck.EEMStatusCheck.execute(Unknown Source)
INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 |  at com.ca.wcc.launcher.statuscheck.StatusTask.run(Unknown Source)
INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 |  at sun.reflect.GeneratedMethodAccessor581.invoke(Unknown Source)
INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 |  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 |  at java.lang.reflect.Method.invoke(Method.java:498)
INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 |  at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:283)
INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 |  at org.springframework.scheduling.support.MethodInvokingRunnable.run(MethodInvokingRunnable.java:69)
INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 |  at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 |  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 |  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 |  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 |  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 |  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 |  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2022/11/06 03:09:38 |    68291 |  at java.lang.Thread.run(Thread.java:827)
INFO   | jvm 1    | 2022/11/06 03:09:41 |    68294 | @rest < status-check-1> []  WARN #EEMStatusCheck                     # Cannot ping the EEM backend ...
INFO   | jvm 1    | 2022/11/06 03:09:54 |    68307 | @rest < status-check-1> []  WARN #EEMStatusCheck                     # Cannot ping the EEM backend ...
INFO   | jvm 1    | 2022/11/06 03:10:07 |    68320 | @rest < status-check-1> []  WARN #EEMStatusCheck                     # Cannot ping the EEM backend ...

 

If they switch back to original JRE, with no other changes, same WCC works fine.  No errors connecting to EEM like above,  WCC can be used normally too.

Environment

Release : 12.0

Cause

Because JRE got upgraded, the Java layer on WCC server was enforcing stronger encryption for the SSL connection into EEM server.   EEM server still allowed TLS v1.0, TLS v1.1  and TLS v1.2, all of them, but defaulted to 1.0.  The Java client layer using which WCC was making EEM calls, they were attempting TLS v.1.2.   And so the problem.

 

1) To identify this we enabled debug on EEM SDK layer on WCC node by making a change to     /opt/CA/WorkloadAutomationAE/wcc/data/config/application/config/resources/eiam.logj4.config         (backup this file so it is easier to restore to default version)

     <logger name="SDK" additivity="false">
                <level value="trace"/>
                <appender-ref ref="SDK" />
        </logger>

 

        <root>
                <priority value="trace" />
                <appender-ref ref="SDK" />
                <!-- <appender-ref ref="Console" /> -->
        </root>

 

2) Save the file,  no need to restart anything

3) Default file to monitor:  /opt/CA/WorkloadAutomationAE/wcc/log/eem/eiam.sdk.java.log

4) Below error showed up in it:

DEBUG 2022-11-17 07:12:00,340 [status-check-1] [com.ca.eiam.SafeContextFactory] getSafeContext - creating new SafeContext
TRACE 2022-11-17 07:12:00,342 [status-check-1] [com.ca.eiam.SafeCache] SafeCache - Enter
TRACE 2022-11-17 07:12:00,342 [status-check-1] [com.ca.eiam.SafeCache] SafeCache - Leave
TRACE 2022-11-17 07:12:00,343 [status-check-1] [com.ca.eiam.poz.PozFactory] PozFactory - Enter
TRACE 2022-11-17 07:12:00,343 [status-check-1] [com.ca.eiam.poz.PozFactory] PozFactory - Leave
TRACE 2022-11-17 07:12:00,343 [status-check-1] [com.ca.eiam.SafeContext] SafeContext - Enter
TRACE 2022-11-17 07:12:00,344 [status-check-1] [com.ca.eiam.SafeContext] SafeContext - Exit
TRACE 2022-11-17 07:12:00,344 [status-check-1] [com.ca.eiam.SafeContext] setBackend - Enter : params[Backend=EEM-Server.Company.com]
TRACE 2022-11-17 07:12:00,344 [status-check-1] [com.ca.eiam.poz.PozFactory] clearPozNodes - Enter
TRACE 2022-11-17 07:12:00,344 [status-check-1] [com.ca.eiam.poz.PozFactory] clearPozNodes - Leave
TRACE 2022-11-17 07:12:00,344 [status-check-1] [com.ca.eiam.poz.PozFactory] addPozNode - Enter : params [poz_node=EEM-Server.Company.com]
DEBUG 2022-11-17 07:12:00,344 [status-check-1] [com.ca.eiam.poz.PozFactory] addPozNode - poz_node=EEM-Server.Company.com, poz_nodes=[EEM-Server.Company.com]
TRACE 2022-11-17 07:12:00,344 [status-check-1] [com.ca.eiam.poz.PozFactory] addPozNode - Leave
TRACE 2022-11-17 07:12:00,344 [status-check-1] [com.ca.eiam.SafeContext] SafeContext.setBackend: setting backend to EEM-Server.Company.com
TRACE 2022-11-17 07:12:00,344 [status-check-1] [com.ca.eiam.SafeContext] setBackend - Getting public key runMethod
TRACE 2022-11-17 07:12:00,346 [status-check-1] [com.ca.eiam.poz.PozFactory] runMethod - Enter : params[Iclient icl,Host=EEM-Server.Company.com,Sponsor=iAuthority,Method=GetPublicKey,List argList]
TRACE 2022-11-17 07:12:00,346 [status-check-1] [Network] RunBatMethod - Enter : params[Host=EEM-Server.Company.com,Method=GetPublicKey,List arglist]
ERROR 2022-11-17 07:12:00,397 [status-check-1] [Network] RunBatMethod - exception occurred calling Iclient runBatMethod
com.ca.itechnology.iclient.IclException: javax.net.ssl.SSLHandshakeException: The server selected protocol version TLS10 is not accepted by client preferences [TLS12]
 at com.ca.itechnology.iclient.IclUtil.httpRequest(IclUtil.java:1544)
 at com.ca.itechnology.iclient.IclMethod.runBatMethod(IclMethod.java:212)
 at com.ca.itechnology.iclient.Iclient.runBatMethod(Iclient.java:580)
 at com.ca.eiam.poz.Network.RunBatMethod(Network.java:320)
 at com.ca.eiam.poz.PozFactory.runMethod(PozFactory.java:2236)
 at com.ca.eiam.SafeContext.setBackend(SafeContext.java:5707)
 at com.ca.eiam.SafeContextFactory.createSafeContext(SafeContextFactory.java:173)
 at com.ca.eiam.SafeContextFactory.getSafeContext(SafeContextFactory.java:123)
 at com.ca.uejm.access.providers.EmbIAMAccessProvider.establishSafeContext(Unknown Source)
 at com.ca.uejm.access.providers.EmbIAMAccessProvider.ping(Unknown Source)
 at com.ca.wcc.launcher.statuscheck.EEMStatusCheck.execute(Unknown Source)
 at com.ca.wcc.launcher.statuscheck.StatusTask.run(Unknown Source)
 at sun.reflect.GeneratedMethodAccessor581.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:283)
 at org.springframework.scheduling.support.MethodInvokingRunnable.run(MethodInvokingRunnable.java:69)
 at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:827)
Caused by: javax.net.ssl.SSLHandshakeException: The server selected protocol version TLS10 is not accepted by client preferences [TLS12]
 at sun.security.ssl.Alert.createSSLException(Alert.java:131)
 at sun.security.ssl.Alert.createSSLException(Alert.java:117)
 at sun.security.ssl.TransportContext.fatal(TransportContext.java:311)
 at sun.security.ssl.TransportContext.fatal(TransportContext.java:267)
 at sun.security.ssl.TransportContext.fatal(TransportContext.java:258)
 at sun.security.ssl.ServerHello$ServerHelloConsumer.onServerHello(ServerHello.java:943)
 at sun.security.ssl.ServerHello$ServerHelloConsumer.consume(ServerHello.java:869)
 at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377)
 at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
 at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
 at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)
 at sun.security.ssl.SSLTransport.decode(SSLTransport.java:152)
 at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1397)
 at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1305)
 at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
 at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:818)
 at sun.security.ssl.SSLSocketImpl.access$200(SSLSocketImpl.java:73)
 at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1180)
 at org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:160)
 at org.apache.http.impl.io.AbstractSessionOutputBuffer.flush(AbstractSessionOutputBuffer.java:168)
 at org.apache.http.impl.io.ContentLengthOutputStream.flush(ContentLengthOutputStream.java:99)
 at org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:115)
 at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:94)
 at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:110)
 at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:118)
 at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:266)
 at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:214)
 at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
 at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
 at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:679)
 at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:481)
 at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
 at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
 at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
 at com.ca.itechnology.iclient.IclUtil.httpRequest(IclUtil.java:1509)
 ... 24 more

Resolution

Enable TLS 1.2 on iGateway

1) backup /opt/CA/SharedComponents/iTechnology/igateway.conf   file

2) edit original /opt/CA/SharedComponents/iTechnology/igateway.conf 

3) look for string   <secureProtocol/>

4) change it to look like :  <secureProtocol>TLSv1_2</secureProtocol>

5) save the file and restart igateway

6) ./S99igateway stop ; sleep 1 ; ./S99igateway start

7) netstat -an |grep 5250       ## to verify the port is now in LISTEN mode

8) verify you can still access EEM URL via a browser now

9) Verify if WCC login works fine now, the errors should disappear

10) restore original  /opt/CA/WorkloadAutomationAE/wcc/data/config/application/config/resources/eiam.logj4.config   file to disable debug.