Overcome Gateway hangs if remote syslog not responding

Products

• STARTER PACK-7
• CA Rapid App Security
• CA API Gateway

Issue/Introduction

At the time of writing this KB, we had a known issue in which the CA API Gateway can hang or not start if Audit Sink to a remote Syslog was set and the remote syslog was not responding in time.

We already alerted our customers about it:

https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=74318

In this KB we will discuss the possible workarounds.

Environment

There are a few possible workaround and we will highlight the 3rd one as it was not discussed so far:

- (1) Install a 3rd party application, such as SPLUNK forwarder and configure the Gateway Audit sink to be written to a local text file, which the SPLUNK forwarder will process and push the events to a remote SPLUNK Server.

- (2) Configure the Audit Sink syslog to use UDP instead of TCP.

  This option have two drawbacks: There is no retry mechanism and if remote syslog server not accessible- data can be lost.

- (3) Configure the Gateway Audit sink to send to 'local' Gateway syslog and then configure the Gateway rsyslog to direct those messages to a remote syslog server.

Resolution

Connect to your Policy Manager and open the Manage Audit Sink task and set the syslog one to send to the local Gateway on a specific port of your choice:

Now, ssh into your Gateway Server and edit the /etc/rsyslog.conf file:

un-comment the below lines and set the TCP port you set in the Policy Manager, Audit Sink to syslog.

#$ModLoad imtcp

#$InputTCPServerRun 1514

Review the rsyslog configuration options and set the needed filters if needed more and then add a line like the below example:

*.* @@myremotesyslog.my.domain:1415

(This can be an IP or a hostname and the port your remote syslog server is listening on)

Save the file and restart the rsyslog by running:

service rsyslog restart

Now, your Gateway is configured to forward to it's own syslog daemon the type of events you set in the Audit Sink and this rsyslog daemon is configured to forward those events to your remote syslog server.

By doing so, the Gateway Service will not hang or stop - as your local Gateway syslog daemon is always accessible for the Gateway and you should be able to proceed with your Audit data sent to your remote syslog for your analysis.

 

Additional Information

This issue was resolved in gateway 9.3 CR5 (and higher), if seeing this issue it is recommended to install the latest patch for your gateway.