Updating index failure error when pushing data to region

Products

• VMware Tanzu Gemfire

Issue/Introduction

When you define a region index by the old and deprecated method (using the functional tag) in Gemfire9.x, you may get the IMQException: "Updating the Index XXXX failed. The index is corrupted and marked as invalid" when pushing the data to region.

For example (Region and Index definition):

<region name="TestRegion" refid="mainReplicateAttributes">
    <region-attributes>
        <value-constraint>com.example.type.TestDomainClass</value-constraint>
    </region-attributes>
    <index name="TestIndexName">
        <functional from-clause="/TestRegion" expression="TestFieldName1"/>
    </index>
</region>

Exception stack in the cacheserver's log:

[warn 2020/09/01 09:14:07.062 AEST <P2P message reader for 10.39.237.214(example_server:62375)<v3>:41001 unshared ordered uid=88 dom #1 port=47232> tid=0x6584] Updating the Index TestIndexName failed. The index is corrupted and marked as invalid.
org.apache.geode.cache.query.internal.index.IMQException
	at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1414)
	at org.apache.geode.cache.query.internal.index.CompactRangeIndex.addMapping(CompactRangeIndex.java:145)
	at org.apache.geode.cache.query.internal.index.AbstractIndex.addIndexMapping(AbstractIndex.java:489)
	at org.apache.geode.cache.query.internal.index.IndexManager.addIndexMapping(IndexManager.java:1152)
	at org.apache.geode.cache.query.internal.index.IndexManager.processAction(IndexManager.java:1060)
	at org.apache.geode.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:985)
	at org.apache.geode.cache.query.internal.index.IndexManager.updateIndexes(IndexManager.java:960)
	at org.apache.geode.internal.cache.LocalRegion.basicPutPart2(LocalRegion.java:5745)
	at org.apache.geode.internal.cache.map.RegionMapPut.doBeforeCompletionActions(RegionMapPut.java:282)
	at org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPutAndDeliverEvent(AbstractRegionMapPut.java:301)
	at org.apache.geode.internal.cache.map.AbstractRegionMapPut.runWithIndexUpdatingInProgress(AbstractRegionMapPut.java:308)
	at org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPutIfPreconditionsSatisified(AbstractRegionMapPut.java:296)
	at org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPutOnSynchronizedRegionEntry(AbstractRegionMapPut.java:282)
	at org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPutOnRegionEntryInMap(AbstractRegionMapPut.java:273)
	at org.apache.geode.internal.cache.map.AbstractRegionMapPut.addRegionEntryToMapAndDoPut(AbstractRegionMapPut.java:251)
	at org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPutRetryingIfNeeded(AbstractRegionMapPut.java:216)
	at org.apache.geode.internal.cache.map.AbstractRegionMapPut.doWithIndexInUpdateMode(AbstractRegionMapPut.java:193)
	at org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPut(AbstractRegionMapPut.java:180)
	at org.apache.geode.internal.cache.map.AbstractRegionMapPut.runWhileLockedForCacheModification(AbstractRegionMapPut.java:119)
	at org.apache.geode.internal.cache.map.RegionMapPut.runWhileLockedForCacheModification(RegionMapPut.java:161)
	at org.apache.geode.internal.cache.map.AbstractRegionMapPut.put(AbstractRegionMapPut.java:169)
	at org.apache.geode.internal.cache.AbstractRegionMap.basicPut(AbstractRegionMap.java:2044)
	at org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5602)
	at org.apache.geode.internal.cache.DistributedRegion.virtualPut(DistributedRegion.java:387)
	at org.apache.geode.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:170)
	at org.apache.geode.internal.cache.LocalRegion.basicUpdate(LocalRegion.java:5573)
	at org.apache.geode.internal.cache.AbstractUpdateOperation.doPutOrCreate(AbstractUpdateOperation.java:156)
	at org.apache.geode.internal.cache.AbstractUpdateOperation$AbstractUpdateMessage.basicOperateOnRegion(AbstractUpdateOperation.java:307)
	at org.apache.geode.internal.cache.AbstractUpdateOperation$AbstractUpdateMessage.operateOnRegion(AbstractUpdateOperation.java:278)
	at org.apache.geode.internal.cache.DistributedCacheOperation$CacheOperationMessage.basicProcess(DistributedCacheOperation.java:1208)
	at org.apache.geode.internal.cache.DistributedCacheOperation$CacheOperationMessage.process(DistributedCacheOperation.java:1110)
	at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:376)
	at org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:431)
	at org.apache.geode.distributed.internal.ClusterDistributionManager.scheduleIncomingMessage(ClusterDistributionManager.java:2071)
	at org.apache.geode.distributed.internal.ClusterDistributionManager.handleIncomingDMsg(ClusterDistributionManager.java:1846)
	at org.apache.geode.distributed.internal.membership.gms.GMSMembership.dispatchMessage(GMSMembership.java:931)
	at org.apache.geode.distributed.internal.membership.gms.GMSMembership.handleOrDeferMessage(GMSMembership.java:862)
	at org.apache.geode.distributed.internal.membership.gms.GMSMembership.processMessage(GMSMembership.java:1204)
	at org.apache.geode.distributed.internal.DistributionImpl$MyDCReceiver.messageReceived(DistributionImpl.java:822)
	at org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:613)
	at org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:661)
	at org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3150)
	at org.apache.geode.internal.tcp.Connection.readMessage(Connection.java:2897)
	at org.apache.geode.internal.tcp.Connection.processInputBuffer(Connection.java:2704)
	at org.apache.geode.internal.tcp.Connection.readMessages(Connection.java:1597)
	at org.apache.geode.internal.tcp.Connection.run(Connection.java:1440)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.geode.security.NotAuthorizedException: Unauthorized access to method: getTestFieldName1
	at org.apache.geode.cache.query.internal.AttributeDescriptor.readReflection(AttributeDescriptor.java:124)
	at org.apache.geode.cache.query.internal.AttributeDescriptor.read(AttributeDescriptor.java:83)
	at org.apache.geode.cache.query.internal.PathUtils.evaluateAttribute(PathUtils.java:77)
	at org.apache.geode.cache.query.internal.CompiledPath.evaluate(CompiledPath.java:149)
	at org.apache.geode.cache.query.internal.CompiledID.evaluate(CompiledID.java:85)
	at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.applyProjection(CompactRangeIndex.java:1557)
	at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1536)
	at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.doNestedIterations(CompactRangeIndex.java:1546)
	at org.apache.geode.cache.query.internal.index.CompactRangeIndex$IMQEvaluator.evaluate(CompactRangeIndex.java:1399)
	... 48 more

Environment

Product Version: 9.1

Resolution

You can confirm the index's status by checking the "Valid Index" field using the "gfsh list indexes" command. "Valid Index"=false indicates that this index is corrupted, which can be caused by the old and deprecated functional index definition.

gfsh>list indexes
Member Name |                Member ID                 | Region Path |     Name      | Type  | Indexed Expression | From Clause | Valid Index
----------- | ---------------------------------------- | ----------- | ------------- | ----- | ------------------ | ----------- | -----------
server1     | 173.16.203.161(server1:138726)<v2>:41002 | /TestRegion | TestIndexName | RANGE | TestFieldName1     | /TestRegion | false
server2     | 173.16.203.161(server2:137065)<v1>:41001 | /TestRegion | TestIndexName | RANGE | TestFieldName1     | /TestRegion | true

You can change the index definition to the new fully supported format to resolve this issue.

For example:

<region name="TestRegion" refid="mainReplicateAttributes">
    <region-attributes>
        <value-constraint>com.example.type.TestDomainClass</value-constraint>
    </region-attributes>
    <index name="TestIndexName" from-clause="/TestRegion" expression="TestFieldName1"  type="range"/>
</region>