Troubleshooting InternalFunctionInvocationTargetException Warnings

Products

• VMware Tanzu Gemfire

Issue/Introduction

Symptoms:
The purpose of this article is to explain why this exception may be seen in the server logs.

Environment

Resolution

The Gemfire client maintains bucket information as metadata, which helps in guiding the function execution to the right node when using single hop operation. However, when a client is newly registered it needs some time to get this metadata information. During this time we can get the below warnings
com.gemstone.gemfire.internal.cache.execute.InternalFunctionInvocationTargetException. This is just a warning as multiple hops are used to execute the function simply because the client is lacking the metadata to know where to execute the function. This warning stops as soon as the client has the required metadata and then the functions start execution with single hop behavior.

The metadata is refreshed during the following events which can result in the same exception logged on the server side with warning log level:

1. Rebalance.
2. The client goes down and comes back again.
3. The data nodes go down or become unresponsive. During this period we can also see the InternalFunctionInvocationTargetException.

The warning can also happen when isHa is set to true in the function code. If function execution on one node fails gemfire will retry the function execution on another node and this exception will be logged.

InternalFunctionInvocationTargetException can also be logged if a client has blacklisted the server where the function should be executed on. Check the client logs to make sure this has not happened.

Lastly, this exception can also happen if there is a problem with the partition resolver or hashcode so the bucket is not consistently resolved.

If there are few connections and many threads doing operations on the client, which can happen if max-connections is too low, then a thread might use an available connection even if it is not to the correct server, thereby getting the InternalFunctionInvocationTargetException.