search cancel

Sudden degraded performance on one gateway in a cluster of 2

book

Article ID: 254176

calendar_today

Updated On:

Products

CA API Gateway

Issue/Introduction

We are recently observing performance issues with one of 2 gateways in our cluster. 

We are logging response times and they only seem to be high on a single node. 

We restarted the services and also rebooted both servers and are still having the issue.  I ran a data collection and have attached it here for your review.

Environment

Release : 10.0

Cause

The Primary SSG DB used by the Gateway is having an issue.  There is a known  bug causing the mysql to crash 

Stack analysis has threads blocked in mysql, the mysql log has the following crash logged 

2022-10-28T19:23:36.593814Z 5 [ERROR] [MY-010584] [Repl] Slave I/O for channel '': error connecting to master '[email protected]:3307' - retry-time: 100 retries: 4 message: Can't connect to MySQL server on 'HOSTNAME' (111), Error_code: MY-002003

19:25:16 UTC - mysqld got signal 11 ;

Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.

Thread pointer: 0x7f0b380008c0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 7f0c08b309c8 thread_stack 0x46000

/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x21e109d]

/usr/sbin/mysqld(handle_fatal_signal+0x313) [0x10096b3]

/lib64/libpthread.so.0(+0xf630) [0x7f0c1ec0d630]

/usr/sbin/mysqld(mysql_compress_context_init(mysql_compress_context*, enum_compression_algorithm, unsigned int)+0) [0x21d9000]

/usr/sbin/mysqld() [0xfe9e05]

Resolution

Recommend updating to the latest PLatform patch  Layer7_API_PlatformUpdate_64bit_v10.X-CentOS-2022-09-22.zip

Link: 

https://support.broadcom.com/web/ecx/support-content-notification/-/external/content/release-announcements/CA-API-Gateway-Solutions--Patches/3024

The current version you are running is:   mysqld 8.0.23-commercial

Newer patches:  '8.0.30-commercial