The /var/log/mysqld.log shows a lot of errors,
lave SQL for channel '': Could not execute Delete_rows event on table otk_db.oauth_token; Can't find record in 'oauth_token', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log ssgbin-log.001233, end_log_pos 427158534, Error_code: 1032
slave status is all right.
Release : 9.3
Component : API GTW ENTERPRISE MANAGER
The error indicates that the master deletes a row but when the slave try to do the same, it cannot find it in its database.
The gateway cluster has multiple gateway nodes and configured to connect to different database, therefore the otk maintenance scheduled tasks are running on multiple gateway nodes, so the expired tokens have been deleted in both DB nodes at the same time, and when the replication tries to delete again, it causes the key not found error.
1. Configure all gateway nodes connect to the same primary database, not localhost
2. The same the otk jdbc connection should be configured to primary database, not localhost
https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=131492
3. Ensure the schedule tasks are configured to run on "One Node"