NSX Intelligence: Redis data directory gets filled up and roll-up doesn’t work as expected
Article ID: 319092
Updated On: 06-05-2023
Products
VMware NSX
Issue/Introduction
Symptoms:
- NSX Application Platform is degraded on NSX Intelligence 3.2.0 or 3.2.0.1 after upgrading from 1.2.x versions.
- Intelligence is showing Down
- Review pod data directory utilization using below command on Redis pods: redis-master-0, redis-slave-0, redis-slave-1
kubectl exec -it -n nsxi-platform redis-slave-0 –df -kh /data
or, from NSX Manager
napp-k exec -it -n nsxi-platform redis-slave-0 –df -kh /data
When this issue is present, the output indicates the data utilization to be near 100%.
- NSX Application Platform is degraded on NSX Intelligence 3.2.0 or 3.2.0.1 after upgrading from 1.2.x versions.
- Intelligence is showing Down
- Review pod data directory utilization using below command on Redis pods: redis-master-0, redis-slave-0, redis-slave-1
kubectl exec -it -n nsxi-platform redis-slave-0 –df -kh /data
or, from NSX Manager
napp-k exec -it -n nsxi-platform redis-slave-0 –df -kh /data
When this issue is present, the output indicates the data utilization to be near 100%.
Environment
VMware NSX-T Data Center
Cause
During an upgrade from NSX Intelligence 1.2.x, a script is used to persist data from druid to Redis. Once the upgrade is complete, latestflow service takes care of maintaining up to one million flows in Redis. When latestflow tries to evict data once it reaches one million flows, it fails with "java.lang.ClassCastException". The data in Redis keeps growing, and rewrite operations get stuck due to the large data size.
Resolution
This issue is resolved in NSX Intelligence 3.2.1 and later releases.
Workaround:
Manually delete the files from redis-master pod and delete the redis-master-0 pod. The pod will start up automatically and the slave pod will be synced to master.
1. Delete the files - appendonly.aof and dump.rdb, in redis-master-0
kubectl exec -it -n nsxi-platform redis-master-0 -- rm /data/appendonly.aof
kubectl exec -it -n nsxi-platform redis-master-0 -- rm /data/dump.rdb
2. Delete the redis-master-0 pod itself
kubectl delete pod -n nsxi-platform redis-master-0
OR, to delete the files through the NSX Manager.
1. Delete the files - appendonly.aof and dump.rdb, in redis-pod
napp-k exec -it redis-master-0 -- rm /data/appendonly.aof
napp-k exec -it redis-master-0 -- rm /data/dump.rdb
2. Delete the redis-master-0 pod itself
napp-k delete pod -n nsxi-platform redis-master-0
Workaround:
Manually delete the files from redis-master pod and delete the redis-master-0 pod. The pod will start up automatically and the slave pod will be synced to master.
1. Delete the files - appendonly.aof and dump.rdb, in redis-master-0
kubectl exec -it -n nsxi-platform redis-master-0 -- rm /data/appendonly.aof
kubectl exec -it -n nsxi-platform redis-master-0 -- rm /data/dump.rdb
2. Delete the redis-master-0 pod itself
kubectl delete pod -n nsxi-platform redis-master-0
OR, to delete the files through the NSX Manager.
1. Delete the files - appendonly.aof and dump.rdb, in redis-pod
napp-k exec -it redis-master-0 -- rm /data/appendonly.aof
napp-k exec -it redis-master-0 -- rm /data/dump.rdb
2. Delete the redis-master-0 pod itself
napp-k delete pod -n nsxi-platform redis-master-0
Additional Information
Impact/Risks:
NSX Application Platform and applications will not function properly and no visualization will be shown for "NOW" period in the Intelligence UI.
NSX Application Platform and applications will not function properly and no visualization will be shown for "NOW" period in the Intelligence UI.
Feedback
Was this article helpful?
Yes
No
Powered by
