Valkey Operator Does Not Restart Pods After Spec Updates Leading to Pending StatefulSet

Products

• VMware Tanzu Data Suite
• VMware Tanzu Greenplum
• VMware Tanzu Greenplum / Gemfire
• VMware Tanzu for Valkey

Issue/Introduction

In Valkey deployments managed by the Valkey Kubernetes Operator, updates made to the ValkeyReplication custom resource (such as removing incorrect node affinity or tolerations) are not automatically applied to running pods.

When a wrong node pool (node affinity) is configured, pods remain in a Pending state due to scheduling constraints. Even after correcting and reapplying the configuration, the pods are not restarted or rescheduled automatically, resulting in continued scheduling failures.

Error Observed:

0/5 nodes are available: 5 node(s) didn't match Pod's node affinity/selector
Pod didn't trigger scale-up

Environment

Valkey Version: 3.3.4

Resolution

Workarounds

1. Manually delete the pods

- The StatefulSet recreates the pod with the updated configuration, allowing successful scheduling.

Example :

kubectl delete pod <pod_name>

 

2. Recreate the ValkeyReplication resource

- Delete and reapply the resource:

kubectl delete -f <file.yaml>
kubectl apply -f <file.yaml>

 

Expected behaviour

When fields that affect pod scheduling or runtime (such as affinity or tolerations) are modified:

• The operator should automatically restart or recreate the pods 
• This ensures that the new configuration is applied without manual intervention

Product team is aware of above limitation and will address the concern in upcoming releases , please keep a watch on release notes.