How to fix java.lang.OutOfMemoryError: Direct buffer error in DATAFLOW and SKIPPER logs
Article ID: 297206
Updated On:
Products
Support Only for Spring
Issue/Introduction
Sometimes, you can get java.lang.OutOfMemoryError: Direct buffer errors in Dataflow and Skipper which can prevent the launch of tasks and streams respectively. One scenario where we have observed this is while launching multiple tasks or streams in sequence within a production environment. The default direct memory in the JVM is 10M, which is inadequate for these use cases. Currently, the tile does not provide you with the ability to configure this value. However, you can increase it to a value that would be appropriate for your app as explained in the next section. You can start with a value of 30M and increase it incrementally, as needed.
Note: The issue affects multiple versions of the SCDF tile. However, there are improvements in memory handling in tile versions 1.10.1 and above. So it is recommended to be on this version or a higher version.
Note: The issue affects multiple versions of the SCDF tile. However, there are improvements in memory handling in tile versions 1.10.1 and above. So it is recommended to be on this version or a higher version.
Environment
Product Version: Other
Resolution
From the user’s space / org where the data flow SI was created, run the following command:
Where dataflowSI is the name of the Dataflow service instance.
You should now be in the org / space of the data flow and skipper backing apps. Next, run the commands below. In this example, I have increased the value to 30m. If the error is in the Dataflow logs, you would execute these commands:
If the error is in the skipper logs, you would execute these commands instead:
If you continue seeing memory errors after increasing the direct buffer memory, you can enable DEBUG level logging as shown below and contact Tanzu Support with the Dataflow and/or Skipper logs.
cf t -o p-dataflow -s $(cf service dataflowSI —guid)
Where dataflowSI is the name of the Dataflow service instance.
You should now be in the org / space of the data flow and skipper backing apps. Next, run the commands below. In this example, I have increased the value to 30m. If the error is in the Dataflow logs, you would execute these commands:
cf set-env dataflow JAVA_OPTS '-XX:MaxDirectMemorySize=30m’ cf restage dataflow
If the error is in the skipper logs, you would execute these commands instead:
cf set-env skipper JAVA_OPTS '-XX:MaxDirectMemorySize=30m’ cf restage skipper
If you continue seeing memory errors after increasing the direct buffer memory, you can enable DEBUG level logging as shown below and contact Tanzu Support with the Dataflow and/or Skipper logs.
$ cf set-env dataflow JAVA_OPTS '-Dlogging.level.cloudfoundry-client=DEBUG -Dlogging.level.reactor.ipc.netty=DEBUG' $ cf restage dataflow
$ cf set-env skipper JAVA_OPTS '-Dlogging.level.cloudfoundry-client=DEBUG -Dlogging.level.reactor.ipc.netty=DEBUG' $ cf restage skipper
Feedback
Yes
No
Powered by
