JedisDataException: OOM command not allowed when used memory > 'maxmemory' in Tanzu Redis

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

Applications connecting to a VMware Tanzu SQL with Redis (or Redis for Tanzu) instance fail with the following exception: redis.clients.jedis.exceptions.JedisDataException: OOM command not allowed when used memory > 'maxmemory'

• The application can still perform read operations.
• All write operations (SET, LPUSH, etc.) are rejected by the Redis server.
• The issue often occurs suddenly in production environments under high load or after a period of data accumulation.

Cause

This error is a self-protection mechanism in Redis. It occurs when the data stored in the Redis instance reaches the maxmemory limit defined by the service plan (e.g., a "Small" plan might have a 1GB limit).

Common root causes include:

1. Data Growth: The volume of data has naturally outgrown the current service plan.
2. Missing TTLs: Keys are being written without a Time-To-Live (TTL), leading to indefinite data accumulation.
3. Eviction Policy: The Redis maxmemory-policy is set to noeviction, which prevents Redis from deleting old keys to make room for new ones when the limit is reached.

Resolution

To resolve the immediate Out-of-Memory (OOM) state and prevent recurrence, follow these steps:

1. Scale the Service Plan

The fastest way to restore write functionality is to increase the available memory by upgrading the service plan.

• Run the following command via the Cloud Foundry CLI: cf update-service <SERVICE_INSTANCE_NAME> -p <NEW_LARGER_PLAN>

2. Identify and Clean Up Data

If scaling is not desired, you must reduce the memory footprint:

• Big Keys: Use the Redis CLI to find large keys: redis-cli --bigkeys. Use MEMORY USAGE <key> to retrieve individual key size.   
• Manual Deletion: Delete unnecessary or stale data using the UNLINK or  DEL command.
• Flush (Caution): If the data is entirely transient/cache, a FLUSHDB can be performed, though this will clear all data in the database.

3. Configure Expiration and Eviction

• Set TTLs: Modify application logic to ensure every key is created with an appropriate expiration time.
• Adjust Eviction Policy: Ensure the Redis instance is configured with an active eviction policy, such as allkeys-lru. This allows Redis to automatically reclaim memory by removing the least recently used keys when it hits the maxmemory limit.

Impact/Risks

• Scaling: Upgrading a service plan may trigger a brief connection blip depending on the service broker implementation and high-availability configuration.
• Eviction: Changing the eviction policy to something other than noeviction may result in the loss of data if it has not been persisted elsewhere.

Additional Information

Related Information

• Redis Documentation: Using Redis as an LRU cache