search cancel

AXA 17.3.2 - Unable to see Session details - "NO DATA" message

book

Article ID: 201096

calendar_today

Updated On:

Products

CA App Experience Analytics

Issue/Introduction

Symptoms

I am unable to see details of the AXA SESSIONS, when I click on any of the sessions to view the details, I get "NO DATA"
It seems to affect only specific data range, for example until August 4th it reports the correct data

TEST#1 - Running a query for data before August 5th ==> OK

 
 
TEST#2 - Running a query for data past 7 days, I can't see details of the SESSION detail as above. Why?
 
 

Also, the following exception is found in the elasticSearch.log:

 see this Exception in the elasticSearch.log:

[2020-10-01T08:50:47,363][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/apphome/aoPlatform/elastic/data/data/CA_AOPlatform-ProdCluster]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.5.jar:5.6.5]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.5.jar:5.6.5]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) ~[elasticsearch-5.6.5.jar:5.6.5]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.5.jar:5.6.5]
    at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-5.6.5.jar:5.6.5]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.6.5.jar:5.6.5]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.6.5.jar:5.6.5]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/apphome/aoPlatform/elastic/data/data/CA_AOPlatform-ProdCluster]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
    at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:261) ~[elasticsearch-5.6.5.jar:5.6.5]
    at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-5.6.5.jar:5.6.5]
    at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-5.6.5.jar:5.6.5]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:233) ~[elasticsearch-5.6.5.jar:5.6.5]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.5.jar:5.6.5]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.5.jar:5.6.5]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.5.jar:5.6.5]
    ... 6 more
    
[2020-10-01T09:31:53,722][DEBUG][o.e.a.s.TransportSearchAction] [vpXbCFy] All shards failed for phase: [query]
[2020-10-01T09:31:54,724][DEBUG][o.e.a.s.TransportSearchAction] [vpXbCFy] All shards failed for phase: [query]    

 

Cause

Some of the Elastic indices were corrupted affecting the display and data ingestion

The data corruption was caused because an unexpected reboot of the server, in this case an OS patch was applied.

 

You can verify indices corruption by checking the status of the indices:

a) Query the indices using query:  {ElasticSearch-endpoint_and_por}/_cat/indices?v

check the health column, look for the ones reporting as "red"

b) You can use the curl command, here is an example:

Environment

Release : 17.3.2

Component : APP EXPERIENCE ANALYTICS ENGINE

Resolution

No available workaround, Elastic indices cannot be repair.

 

It is strongly recommended to stop all axa services using the AXA-HOME/bin/stopServices.sh before a server reboot to prevent a data corruption.

 

Alternatively, you can try to restart all AXA services AXA-HOME/bin/stopServices.sh and startServices.sh, make sure all services report as running, below an example:

======================== All Components Health =========================
Thu Oct  1 11:39:07 CEST 2020
Aggregator                                       --RUNNING-->      108516
Transformer                                      --RUNNING-->      108583
Crash Handler                                    --RUNNING-->      108648
DxC (Collector)                                  --RUNNING-->      108710
Combined Read and Admin UI TomEE Instance        --RUNNING-->      107283
Jarvis Onboarding Tomcat Instance                --RUNNING-->      106772
Jarvis Indexer                                   --RUNNING-->      107081
Jarvis Verifier                                  --RUNNING-->      107082
Jarvis Schema Registry                           --RUNNING-->      107076
Jarvis Utils                                     --RUNNING-->      107084
Elasticsearch Server                             --RUNNING-->      4304
Kafka Server                                     --RUNNING-->      4057
Zookeeper Server                                 --RUNNING-->      2651
Kibana Tenant Manager                            --RUNNING-->      108781
LogAnalytics Log-collector                       --RUNNING-->      108874
LogAnalytics Log-parser                          --RUNNING-->      109129
Spark Master                                     --RUNNING-->      109459
Spark Worker                                     --RUNNING-->      109627
Spark Scheduler                                  --RUNNING-->      109772
APM Logstash Plugin                              --RUNNING-->      109871
========================================================================

 

Attachments