Cannot connect to WCC

Products

• CA Workload Automation AE - Scheduler (AutoSys)

Issue/Introduction

Cannot connect to WCC. On and off, the CA WCC services go down.

Environment

Release : 12.0

Component :

Cause

In CA-wcc.log, noticed WCC server took 108 seconds to start up and then exited unexpected:

INFO   | jvm 1    | 2021/11/09 03:28:46 |      108 | INFO: Server startup in [108,684] milliseconds
DEBUG  | wrapperp | 2021/11/09 03:28:46 |      108 | socket read failed. (An existing connection was forcibly closed by the remote host. (0x2746))
DEBUG  | wrapperp | 2021/11/09 03:28:46 |      108 | Closing backend socket.
DEBUG  | wrapperp | 2021/11/09 03:28:46 |      108 | server listening on port 32002.
DEBUG  | wrapper  | 2021/11/09 03:28:47 |      109 | JVM process exited with a code of 0, leaving the Wrapper exit code set to 0.
ERROR  | wrapper  | 2021/11/09 03:28:47 |      109 | JVM exited unexpectedly.
DEBUG  | wrapper  | 2021/11/09 03:28:47 |      109 | Preparing to restart with mode 2.
DEBUG  | wrapper  | 2021/11/09 03:28:47 |      109 | JVM was only running for 108 seconds leading to a failed restart count of 1.
DEBUG  | wrapper  | 2021/11/09 03:28:47 |      109 | Waiting 5 seconds before launching another JVM.

Note that socket connection was forcible closed by remote host.

JVM restarted and ran into the same issue again.  

It repeated five times before giving up eventuallly.

INFO   | jvm 9    | 2021/11/09 03:47:49 |     1212 | INFO: Server startup in [113,076] milliseconds
DEBUG  | wrapperp | 2021/11/09 03:47:49 |     1212 | socket read failed. (An existing connection was forcibly closed by the remote host. (0x2746))
DEBUG  | wrapperp | 2021/11/09 03:47:49 |     1212 | Closing backend socket.
DEBUG  | wrapperp | 2021/11/09 03:47:49 |     1212 | server listening on port 32002.
DEBUG  | wrapper  | 2021/11/09 03:47:49 |     1212 | JVM process exited with a code of 0, leaving the Wrapper exit code set to 0.
ERROR  | wrapper  | 2021/11/09 03:47:49 |     1212 | JVM exited unexpectedly.
DEBUG  | wrapper  | 2021/11/09 03:47:49 |     1212 | Preparing to restart with mode 2.
DEBUG  | wrapper  | 2021/11/09 03:47:49 |     1212 | JVM was only running for 116 seconds leading to a failed restart count of 5.
FATAL  | wrapper  | 2021/11/09 03:47:49 |     1212 | There were 5 failed launches in a row, each lasting less than 300 seconds.  Giving up.
FATAL  | wrapper  | 2021/11/09 03:47:49 |     1212 |   There may be a configuration problem: please check the logs.
DEBUG  | wrapperp | 2021/11/09 03:47:49 |     1212 | closing backend server.
STATUS | wrapper  | 2021/11/09 03:47:49 |     1212 | <-- Wrapper Stopped

In CA-wcc-services.log, we can see that the connection between CA-wcc and CA-wcc-services (collector) was closed as well:

INFO   | jvm 1    | 2021/11/09 03:47:49 |     3367 | @collector hz.1 WCC_COLLECTOR(hostname.company.com:7004).IO.thread-in-2 []  WARN #TcpIpConnection                    # [hostname.company.com]:7004 [eBrrOwddFj] [3.9.1] Connection[id=15, /10.1.2.3:7004->/10.1.2.3:51577, endpoint=[10.1.2.3]:51577, alive=false, type=JAVA_CLIENT] closed. Reason: Exception in Connection[id=15, /10.1.2.3:7004->/10.1.2.3:51577, endpoint=[10.1.2.3]:51577, alive=true, type=JAVA_CLIENT], thread=hz.1 WCC_COLLECTOR(hostname.company.com:7004).IO.thread-in-2
INFO   | jvm 1    | 2021/11/09 03:47:49 |     3367 | java.io.IOException: An existing connection was forcibly closed by the remote host.
INFO   | jvm 1    | 2021/11/09 03:47:49 |     3367 |  at sun.nio.ch.SocketDispatcher.read0(Native Method)
INFO   | jvm 1    | 2021/11/09 03:47:49 |     3367 |  at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
INFO   | jvm 1    | 2021/11/09 03:47:49 |     3367 |  at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
INFO   | jvm 1    | 2021/11/09 03:47:49 |     3367 |  at sun.nio.ch.IOUtil.read(IOUtil.java:197)
INFO   | jvm 1    | 2021/11/09 03:47:49 |     3367 |  at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:377)
INFO   | jvm 1    | 2021/11/09 03:47:49 |     3367 |  at com.hazelcast.internal.networking.AbstractChannel.read(AbstractChannel.java:94)
INFO   | jvm 1    | 2021/11/09 03:47:49 |     3367 |  at com.hazelcast.internal.networking.nio.NioChannelReader.handle(NioChannelReader.java:127)
INFO   | jvm 1    | 2021/11/09 03:47:49 |     3367 |  at com.hazelcast.internal.networking.nio.NioThread.handleSelectionKey(NioThread.java:401)
INFO   | jvm 1    | 2021/11/09 03:47:49 |     3367 |  at com.hazelcast.internal.networking.nio.NioThread.handleSelectionKeys(NioThread.java:386)
INFO   | jvm 1    | 2021/11/09 03:47:49 |     3367 |  at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:293)
INFO   | jvm 1    | 2021/11/09 03:47:49 |     3367 |  at com.hazelcast.internal.networking.nio.NioThread.run(NioThread.java:248)

Resolution

Shut down anti-virus services, Trend Micro Deep Security Agent and Trend Micro Deep Security Monitor, running on the WCC server.