After changing elastic disk paths 3 pods started to crash after restart
search cancel

After changing elastic disk paths 3 pods started to crash after restart


Article ID: 247612


Updated On:


DX Application Performance Management


We changed our nfs path configuration elasticsearch.

After starting all system jarvis-verifier-5d995889d4-jf5f8,  deployment jarvis-lean-jarvis-indexer, jarvis-indexer-77654b46d9-99gjz is crashing. We cannot login DX platform.


In the jarvis indexer log, we see this error

SLF4J: See for further details.
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 1



Release : 21.3

Component : Introscope


Customer has reconfigured NFS path to add more disk spaces. Restarted the cluster afterwards.

Then they found multiple jarvis pods keep crashing with "ArrayIndexOutOfBoundsException' error, including jarvis-indexer, jarvis-verifier, kron, etc. And pretty much all doi pods in Init state.

We finally figured out the jarvis ConfigMap has ES entries that are obviously wrong, containng empty elements and duplicated elements. Most likely it's not updated properly when removing/adding warm nodes to reconfigure NFS.

kubectl edit cm jarvis-configmap -ndxi

Remove duplicate from 

jarvis-elasticsearch:9300, jarvis-elasticsearch-2:9300, jarvis-elasticsearch-3:9300,,, jarvis-elasticsearch-warm1:9300, jarvis-elasticsearch-warm2:9300 jarvis-elasticsearch-warm1:9300, jarvis-elasticsearch-warm2:9300 jarvis-elasticsearch-warm1:9300, jarvis-elasticsearch-warm2:9300


jarvis-elasticsearch:9300, jarvis-elasticsearch-2:9300, jarvis-elasticsearch-3:9300, jarvis-elasticsearch-warm1:9300, jarvis-elasticsearch-warm2:9300

After fixing the CM, and restart impacted pods, all is good. Verified AXA and DOI UI shows up OK with data.



Additional Information

The file containing the info is jarvis_cm