DiskAccessException - Fatal error from asynchronous flusher thread
Article ID: 294423
Updated On:
Products
Issue/Introduction
Gemfire node stops due to a fatal DiskAccessException from the asynchronous flusher thread caused by ConcurrentModificationException.
This article provides an explanation for how a ConcurrentModificationException can create a DiskAccessException and discusses options to prevent this situation. The entire exception stacktrace logged in the Gemfire servers is captured below.
[ERROR][Monitor Thread-2] Error when collect information from [Region Size] org.apache.geode.cache.CacheClosedException: For DiskStore: _SYSTEM_LOCAL: Fatal error from asynchronous flusher thread ------------------------- Caused by: org.apache.geode.cache.DiskAccessException: For DiskStore: _SYSTEM_LOCAL: Fatal error from asynchronous flusher thread at org.apache.geode.internal.cache.DiskStoreImpl$FlusherThread.doAsyncFlush(DiskStoreImpl.java:1808) ~[geode-core-9.10.15.jar:?] at org.apache.geode.internal.cache.DiskStoreImpl$FlusherThread.run(DiskStoreImpl.java:1722) ~[geode-core-9.10.15.jar:?] ... 1 more Caused by: org.apache.geode.ToDataException: toData failed on DataSerializer with id=0 for class class java.util.HashMap --------------------------- Caused by: java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429) at java.util.HashMap$EntryIterator.next(HashMap.java:1463) at java.util.HashMap$EntryIterator.next(HashMap.java:1461) at org.apache.geode.DataSerializer.writeHashMap(DataSerializer.java:2234) at org.apache.geode.internal.InternalDataSerializer$34.toData(InternalDataSerializer.java:802) at org.apache.geode.internal.InternalDataSerializer.writeUserObject(InternalDataSerializer.java:1568) ... 21 more ----------------------------
Environment
Cause
"java.util.ConcurrentModificationException" is thrown by a method when it detects an object being modified concurrently, when it is not permissible to do so. For example, in Java, a collection cannot be modified by the same or another thread when it is being iterated over. The iterator follows a fail fast approach to prevent non-deterministic values in the collection.
In this case, the “java.util.ConcurrentModificationException” is caused when a referenced object is simultaneously modified/accessed by code that's directly executed on the server, like cache writers and listeners, functions and transactions. A client executes on a different JVM from the server and cannot access an object by reference.
When you allow a referenced object to be modified in this manner, it can also result in cache inconsistency. To avoid this, you can create a copy of the object in one of 2 ways discussed below.
Resolution
Option 1
Enable the copy-on-read parameter through either configuration or code:
- Using cache.xml:
<cache copy-on-read="true">
- or by using the GemFire Java API:
Cache c = CacheFactory.getInstance(system)
c.setCopyOnRead(true);
Impact of enabling copy-on-read parameter :
When copy-on-read is true, all entry access methods return copies of the entries. This protects all server-side code from inadvertently modifying in-place. This attribute will negatively impact performance and memory consumption when a copy is not needed. Note that the copy-on-read attribute impacts the entire cache and cannot be selectively applied to regions.
Option 2
Implement server-side code that creates and uses a copy of the returned object. For objects that are cloneable or serializable, copy the entry value to a new object using org.apache.geode.CopyHelper.copy.
Example:
Object o = region.get(key); StringBuffer s = (StringBuffer) CopyHelper.copy(o);
// further operations on the region entry value will use s
s.toUpperCase();
Always use a Region method to then change data in the region. Do not use the reference returned from the entry access method. If the upper case string should become the new value for the region entry:
region.put(key, s);
Both options are explained in the official documentation linked below.
Feedback
