This can happen for various reasons. It usually happens if the OS has some memory problems. Please try the following:
On Windows, edit file "startServer.lax" on the following line (value in bytes):
On Unix, edit file "startServer" on the following line (value in MegaBytes):
Note: The procedure applies to all UNIX flavors except AIX. we have a separate knowledge document for AIX.