Overcome Gateway hangs if remote syslog not responding

Article Id: 130079
Status: Published
Updated On: 13-02-2020 16:39
Products:
STARTER PACK-7 , CA Rapid App Security , CA API Gateway , CA API Gateway , CA API Gateway
Issue/Introduction:

At the time of writing this KB, we had a known issue in which the CA API Gateway can hang or not start if Audit Sink to a remote Syslog was set and the remote syslog was not responding in time.


We already alerted our customers about it:

https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=74318

In this KB we will discuss the possible workarounds.


Environment:

There are a few possible workaround and we will highlight the 3rd one as it was not discussed so far:

- (1) Install a 3rd party application, such as SPLUNK forwarder and configure the Gateway Audit sink to be written to a local text file, which the SPLUNK forwarder will process and push the events to a remote SPLUNK Server.

- (2) Configure the Audit Sink syslog to use UDP instead of TCP.

  This option have two drawbacks: There is no retry mechanism and if remote syslog server not accessible- data can be lost.

- (3) Configure the Gateway Audit sink to send to 'local' Gateway syslog and then configure the Gateway rsyslog to direct those messages to a remote syslog server.

Resolution:

Connect to your Policy Manager and open the Manage Audit Sink task and set the syslog one to send to the local Gateway on a specific port of your choice:




Manage Log Sinks

Configure the syslof sink

Now, ssh into your Gateway Server and edit the /etc/rsyslog.conf file:

un-comment the below lines and set the TCP port you set in the Policy Manager, Audit Sink to syslog.


#$ModLoad imtcp

#$InputTCPServerRun 1514


Review the rsyslog configuration options and set the needed filters if needed more and then add a line like the below example:


*.* @@myremotesyslog.my.domain:1415


(This can be an IP or a hostname and the port your remote syslog server is listening on)


Save the file and restart the rsyslog by running:


service rsyslog restart


Now, your Gateway is configured to forward to it's own syslog daemon the type of events you set in the Audit Sink and this rsyslog daemon is configured to forward those events to your remote syslog server.


By doing so, the Gateway Service will not hang or stop - as your local Gateway syslog daemon is always accessible for the Gateway and you should be able to proceed with your Audit data sent to your remote syslog for your analysis.

 

Additional Information:

This issue was resolved in gateway 9.3 CR5 (and higher), if seeing this issue it is recommended to install the latest patch for your gateway.