DiskAccessException thrown when interrupting thread modifying an OpLog

Products

• VMware Tanzu Gemfire

Issue/Introduction

Environment: Pivotal GemFire - any version (observed in v8.2)

Please suppose that you have your own logic at peers or cache servers side with the following conditions.

• Configuring disk stores and updating OpLog files on demand in terms of data persisted to disks.
• Managing application threads using thread libraries in your own logic involving thread interruption.

You may see the following DiskAccessException at one of the peers or cache servers unexpectedly, causing them to be automatically shut down if one of your application threads of the peer or cache server, while modifying an OpLog file, is interrupted by another application thread. This could be via some (via Thread#interrupt() call, or by using ExecutorService#shutdownNow() and so on, both which involve immediate thread interruption.

Resolution

Exception stack trace:

[error 2017/07/21 11:46:07.951 JST server1 <Thread-12> tid=0x2a] A DiskAccessException has occurred while writing to the disk for region /__PR/_B__MyRegion_101. The cache will be closed.

com.gemstone.gemfire.cache.DiskAccessException: For Region: _B__MyRegion_101: Failed writing key to "/path/to/persist/OVERFLOWDEFAULT_1"

 at com.gemstone.gemfire.internal.cache.OverflowOplog.modify(OverflowOplog.java:580) ~[gemfire.jar:?]

 at com.gemstone.gemfire.internal.cache.OverflowOplogSet.modify(OverflowOplogSet.java:57) ~[gemfire.jar:?]

 at com.gemstone.gemfire.internal.cache.DiskStoreImpl.put(DiskStoreImpl.java:741) ~[gemfire.jar:?]

 at com.gemstone.gemfire.internal.cache.DiskRegion.put(DiskRegion.java:374) ~[gemfire.jar:?]

 at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.writeToDisk(DiskEntry.java:537) ~[gemfire.jar:?]

 at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.writeToDisk(DiskEntry.java:461) ~[gemfire.jar:?]

 at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.overflowToDisk(DiskEntry.java:1088) ~[gemfire.jar:?]

 at com.gemstone.gemfire.internal.cache.AbstractLRURegionMap.evictEntry(AbstractLRURegionMap.java:287) ~[gemfire.jar:?]

 at com.gemstone.gemfire.internal.cache.AbstractLRURegionMap.lruUpdateCallback(AbstractLRURegionMap.java:495) ~[gemfire.jar:?]

       :

       :

 at java.util.concurrent.RecursiveTask.exec(RecursiveTask.java:94) [?:1.8.0_121]

 at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:1.8.0_121]

 at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) [?:1.8.0_121]

 at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) [?:1.8.0_121]

 at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) [?:1.8.0_121]

Caused by: java.nio.channels.ClosedByInterruptException

 at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) ~[?:1.8.0_121]

 at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:269) ~[?:1.8.0_121]

 at com.gemstone.gemfire.internal.cache.OverflowOplog.writeOpLogBytes(OverflowOplog.java:814) ~[gemfire.jar:?]

 at com.gemstone.gemfire.internal.cache.OverflowOplog.basicModify(OverflowOplog.java:629) ~[gemfire.jar:?]

 at com.gemstone.gemfire.internal.cache.OverflowOplog.modify(OverflowOplog.java:578) ~[gemfire.jar:?]

 ... 25 more

Your application thread may trigger an update to an OpLog file while doing some region operations in the thread locally. The OpLog file is updated using NIO technology. If one of the threads executing NIO operations receives immediate thread interruption, it is terminated and throws a ClosedByInterruptException, according to the API specification of NIO. This is the root cause of this DiskAccessException.

 

Basically, you should not interrupt threads using Thread#interrupt in your application running on a GemFire cache process. If you use the ExecutorService library to manage multiple application threads to run your own tasks and want to interrupt those managed threads for the purpose of stopping your application tasks running on your application threads, please use ExecutorService#shutdown() rather than ExecutorService#shutdownNow().