MessageTooLongException/ MessageTooLargeException
search cancel

MessageTooLongException/ MessageTooLargeException

book

Article ID: 294416

calendar_today

Updated On:

Products

VMware Tanzu Gemfire

Issue/Introduction

GemFire fails to send messages due to MessageTooLargeException with log entries like:

Caused by: org.apache.geode.internal.cache.tier.sockets.MessageTooLargeException: Message size (1742175963) exceeds gemfire.client.max-message-size setting (1073741824)

GemFire fails to send message due to MessageTooLargeException.

Symptoms:

There are many places where a user may see this exception. For instance,

1) On a client when receiving results from a function execution:

org.apache.geode.cache.execute.FunctionException: org.apache.geode.cache.execute.FunctionException: IOException while sending the last chunk to client
    at ------
    at org.apache.geode.internal.cache.execute.AbstractExecution.executeFunctionLocally(AbstractExecution.java:328)
    at org.apache.geode.internal.cache.execute.AbstractExecution.executeFunctionOnLocalPRNode(AbstractExecution.java:263)
    at org.apache.geode.internal.cache.PartitionedRegion.executeOnSingleNode(PartitionedRegion.java:3828)
    at org.apache.geode.internal.cache.PartitionedRegion.executeFunction(PartitionedRegion.java:3533)
    at org.apache.geode.internal.cache.execute.PartitionedRegionFunctionExecutor.executeFunction(PartitionedRegionFunctionExecutor.java:220)
    at org.apache.geode.internal.cache.execute.AbstractExecution.execute(AbstractExecution.java:376)
    at org.apache.geode.internal.cache.execute.AbstractExecution.execute(AbstractExecution.java:359)
    at org.apache.geode.internal.cache.tier.sockets.command.ExecuteRegionFunctionSingleHop.cmdExecute(ExecuteRegionFunctionSingleHop.java:261)
    at org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:183)
    at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMessage(ServerConnection.java:848)
    at org.apache.geode.internal.cache.tier.sockets.OriginalServerConnection.doOneMessage(OriginalServerConnection.java:72)
    at org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1212)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.geode.internal.cache.tier.sockets.AcceptorImpl.lambda$initializeServerConnectionThreadPool$3(AcceptorImpl.java:676)
    at org.apache.geode.logging.internal.executors.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:119)
    at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.geode.cache.execute.FunctionException: IOException while sending the last chunk to client
    at org.apache.geode.internal.cache.execute.ServerToClientFunctionResultSender65.lastResult(ServerToClientFunctionResultSender65.java:164)
    at org.apache.geode.internal.cache.execute.PartitionedRegionFunctionResultSender.lastClientSend(PartitionedRegionFunctionResultSender.java:377)
    at org.apache.geode.internal.cache.execute.PartitionedRegionFunctionResultSender.lastResult(PartitionedRegionFunctionResultSender.java:157)
    ... 19 common frames omitted
Caused by: org.apache.geode.internal.cache.tier.sockets.MessageTooLargeException: Message size (1349791002) exceeds gemfire.client.max-message-size setting (1073741824)
    at org.apache.geode.internal.cache.tier.sockets.Message.sendBytes(Message.java:605)
    at org.apache.geode.internal.cache.tier.sockets.ChunkedMessage.sendChunk(ChunkedMessage.java:312)
    at org.apache.geode.internal.cache.tier.sockets.ChunkedMessage.sendChunk(ChunkedMessage.java:322)
    at org.apache.geode.internal.cache.execute.ServerToClientFunctionResultSender65.lastResult(ServerToClientFunctionResultSender65.java:158)
    ... 22 common frames omitted

2) On a cache server when returning results:

[error 2022/10/03  GMT <Function Execution Processor307> tid=0x90c73f] Error while executing function. 
org.apache.geode.cache.execute.FunctionException: IOException while sending the last chunk to client
    at org.apache.geode.internal.cache.execute.ServerToClientFunctionResultSender65.lastResult(ServerToClientFunctionResultSender65.java:164)
    at org.apache.geode.internal.cache.execute.PartitionedRegionFunctionResultSender.lastClientSend(PartitionedRegionFunctionResultSender.java:377)
    at org.apache.geode.internal.cache.execute.PartitionedRegionFunctionResultSender.lastResult(PartitionedRegionFunctionResultSender.java:157)
    ----
    at org.apache.geode.internal.cache.execute.AbstractExecution.executeFunctionLocally(AbstractExecution.java:328)
    at org.apache.geode.internal.cache.execute.AbstractExecution.lambda$executeFunctionOnLocalPRNode$0(AbstractExecution.java:273)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.geode.distributed.internal.ClusterOperationExecutors.runUntilShutdown(ClusterOperationExecutors.java:441)
    at org.apache.geode.distributed.internal.ClusterOperationExecutors.doFunctionExecutionThread(ClusterOperationExecutors.java:376)
    at org.apache.geode.logging.internal.executors.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:119)
    at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.geode.internal.cache.tier.sockets.MessageTooLargeException: Message size (1160019252) exceeds gemfire.client.max-message-size setting (1073741824)
    at org.apache.geode.internal.cache.tier.sockets.Message.sendBytes(Message.java:605)
    at org.apache.geode.internal.cache.tier.sockets.ChunkedMessage.sendChunk(ChunkedMessage.java:312)
    at org.apache.geode.internal.cache.tier.sockets.ChunkedMessage.sendChunk(ChunkedMessage.java:322)
    at org.apache.geode.internal.cache.execute.ServerToClientFunctionResultSender65.lastResult(ServerToClientFunctionResultSender65.java:158)
    ... 13 more

3) Or, on a client doing a very large putAll:

org.apache.geode.GemFireIOException: unable to transmit message to server
    at org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:178)
    at org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:130)
    at org.apache.geode.cache.client.internal.PoolImpl.execute(PoolImpl.java:792)
    at org.apache.geode.cache.client.internal.PutAllOp.execute(PutAllOp.java:100)
    at org.apache.geode.cache.client.internal.ServerRegionProxy.putAll(ServerRegionProxy.java:607)
    at org.apache.geode.internal.cache.LocalRegion.basicPutAll(LocalRegion.java:9361)
    at org.apache.geode.internal.cache.LocalRegion.putAll(LocalRegion.java:9294)
    at org.apache.geode.internal.cache.LocalRegion.putAll(LocalRegion.java:9306)
    ....
Caused by: org.apache.geode.internal.cache.tier.sockets.MessageTooLargeException: Message size (1645984840) exceeds gemfire.client.max-message-size setting (1073741824)
    at org.apache.geode.internal.cache.tier.sockets.Message.sendBytes(Message.java:599)
    at org.apache.geode.internal.cache.tier.sockets.Message.send(Message.java:1122)
    at org.apache.geode.cache.client.internal.AbstractOp.sendMessage(AbstractOp.java:137)
    at org.apache.geode.cache.client.internal.AbstractOp.attemptSend(AbstractOp.java:91)
    at org.apache.geode.cache.client.internal.AbstractOp.attempt(AbstractOp.java:380)
    at org.apache.geode.cache.client.internal.ConnectionImpl.execute(ConnectionImpl.java:289)
    at org.apache.geode.cache.client.internal.pooling.PooledConnection.execute(PooledConnection.java:338)
    at org.apache.geode.cache.client.internal.OpExecutorImpl.executeWithPossibleReAuthentication(OpExecutorImpl.java:912)
    at org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:174)
    ... 20 more

Environment

Product Version: 9.10

Cause

In all cases, the underlying cause is the same, the sending member (client or server/peer) is trying to send a message that is larger than maximum size set by the parameter gemfire.client.max-message-size. This parameter, which defaults to 1G (1073741824 bytes), is only checked in the send method of the message class so this limit is always determined by the *sender* and the exception will be seen on the *sender*. (In first two examples above, the exception occurs on the cache server when it is trying to return results the client receives the *Exception*, which is what is shown in the first example.)

Resolution

There are really only two options to resolve this issue. The first, which is *not recommended* as it can seriously impact cluster performance and health, is to set the gemfire.client.max-message-size to something larger than its default of 1073741824 bytes.

The second is to rework client code to make the messages smaller. For instance, in the first two exception stacks above, the problem is a function trying to send too big a reply to the invoking client. This could be addresses with some code refactoring and/or better use of a ResultSender. In the third case above, the client is doing a very large putAll. This problem can only really be solved by limiting the numbers of entries in the putAll. Note, however, that similar putAll throughput should be achievable by using multiple threads with smaller the putAll batch sizes.


Powered by Wolken Software