Frequent Backup can Lead to Too Many .crf, .krf Files in the Disk Stores

Products

• VMware Tanzu Gemfire

Issue/Introduction

Symptoms:

Too many .crf and .krf files in the disk directories are causing server start time to increase. And, the backups are very frequent.

Environment

Cause

There are so many files in the disk directories due to the combination of having a lot of disk stores and doing frequent backups. Whenever a backup occurs, we roll to a new oplog, even if the current oplog is nowhere near it's maxed size. Compaction removes these small oplogs if they are empty or mostly garbage, but if they have enough live entries in them, they just sit around. If each small oplog is alive, it ends up with a few live entries because of the updates that happened to all of the disk stores in between the backups.

Resolution

The workaround is to reduce the number of disk stores or reduce the frequency of backups. Another solution would be to do an offline compaction that removes all the garbage and probably consolidates some of the files. But, that would require bringing the member down.