Member crashed after upgrading GemFire 7.x to 8.x when disk store size is limited through ‘dir-size’ attribute.

Products

• VMware Tanzu Gemfire

Issue/Introduction

This article applies to all 8.x versions of GemFire.

To describe a known issue with GemFire startup after and upgrading to GemFire 8 caused by a change in how GemFire manages disk store size limits.

The member crashes after upgrading GemFire 7.x to 8.x when disk store size is limited through the dir-size attribute.

Resolution

This pre-allocation is enabled by default in both GemFire 7.x and 8.x, but the stricter checking of the disk store size, specified by dir-size, is only implemented in GemFire 8.x. The implementation in GemFire 8.x prevents a disk store from growing beyond the specified size causing the server to shutdown and be removed from the distributed system. The same disk store configuration in GemFire 7.x allows the disk store to grow beyond its defined size with only a warning message.

The solution in GemFire 8.x is to:

1. remove the dir-size attribute from the <disk-dir> element when upgrading from 7.x to 8.x. This will allow the disk store to grow up to the available space on the disk or a maximum of 2 Petabytes (i.e. Integer.MAX_VALUE);

2. keep the dir-size attribute aware that the member will crash when the oplog files consume the specified disk store size; or,

3. disable pre-allocation using the command line option --J=-Dgemfire.preAllocateDisk=false to avoid the issue (not recommended when the file system supports pre-allocation).

Example disk store configuration:

<disk-store name="DataPolicyDS" auto-compact="true" max-oplog-size="28">
   <disk-dirs>
        <disk-dir dir-size="112">/opt/gemFire/DataPolicy/data</disk-dir>
   </disk-dirs>
</disk-store>

In GemFire 7.x, when pre-allocation is enabled, GemFire allows the disk store size to grow beyond its specified size, if needed, as part of oplog file rolling and gives following warning message:

[warning 2015/10/15 15:51:36.868 PDT server2 <ServerConnection on port 40406 Thread 1> tid=0x4a] Even though the configured directory size limit has been exceeded a new oplog will be created because compaction is enabled. The configured limit is 117,440,512. The current space used in the directory by this disk store is 422,799,992.

In GemFire 8.x, when pre-allocation is enabled, GemFire restricts the disk store size to the defined size. The oplog rolling will continue as long as it can pre-allocate space for a new oplog file within the directory's capacity. However, upon hitting the disk store directory size, the member will throw the below exception before it is shut down and removed from the cluster:

[error 2015/10/16 11:25:44.941 PDT server2 <ServerConnection on port 58002 Thread 0> tid=0x48] A DiskAccessException has occurred while writing to the disk for region /__PR/_B__Person2_87. The cache will be closed.
com.gemstone.gemfire.internal.InsufficientDiskSpaceException: For DiskStore: DataPolicyDS: Could not pre-allocate file /opt/gemFire/DataPolicy/data/BACKUPDataPolicyDS_5.crf with size=26,424,115, caused by java.io.IOException: not enough space left to pre-blow, available=11741090, required=26424115
at com.gemstone.gemfire.internal.cache.Oplog.preblow(Oplog.java:1063)

In GemFire 8.x, when pre-allocation is disabled using the system property preAllocateDisk=false, GemFire allows the disk store to grow beyond its defined size with same warning given by GemFire 7.x:

[warning 2015/10/15 17:06:53.824 PDT server2 <ServerConnection on port 58002 Thread 0> tid=0x48] 

Even though the configured directory size limit has been exceeded a new oplog will be created because compaction is enabled. The configured limit is 117,440,512. The current space used in the directory by this disk store is 237,824,998.