In most situations, this error message is not related to a memory issue, but results from a configuration error.
One of possible reasons is a security policy containing an international or non-ASCII character.
As a workaround, you can disable "rulebase_uids_in_log" parameter, or contact Check Point to obtain a hotfix.
Go to 'Policy' menu - Global Properties - SmartDashboard Customization - under Advanced Configuration, click on 'Configure...' - expand the 'FireWall-1' - go to 'General'.
Clear the box rulebase_uids_in_log.
Click 'OK' to close all windows.
Install the Security policy.
Another possibility is a cluster IP address configured in "General properties" and conflicting with one of the IP addresses on the gateway.