Production CA-wcc-services Server service is going down frequently which is impacting business.

Products

• Workload Automation Agent

Issue/Introduction

Its been observed that only the production Wcc service server going down frequently and we are getting error page as "500" and sometimes its not getting connected.

               CA Services Status Report

           Component Name               Pid        Status
------------------------------------  -------  --------------
WAAE Agent (WA_AGENT_PRD)                1624  running
CA-wcc-services Server                  
CA-wcc-db Server                        40072  running
CA-wcc Server                           57497  running

 

 

Environment

Release : 11.3.6

Component :

Resolution

We can see this error each time when the problem happened.

CA-wcc-services.log 

INFO   | jvm 1    | 2022/05/31 12:50:34 |    76000 | INFO: Error parsing HTTP request header
INFO   | jvm 1    | 2022/05/31 12:50:34 |    76000 |  Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
INFO   | jvm 1    | 2022/05/31 12:50:34 |    76000 | java.lang.IllegalArgumentException: Invalid character found in method name. HTTP method names must be tokens

It appears to be receiving a request that is not properly formatted. This could be caused due to some port scanners or antivirus scanners that seems to be sending this.

Finally, it was  found that splunkd service re-start was stopping Autosys WCC, currently we have disabled/stopped splunkd service from re-start and Autosys wcc service is up and running