AIOps - Unable to query the Elastic snapshots - repository_exception "Could not read repository data because the contents of the repository do not match its expected state"

Products

• DX Operational Intelligence
• DX Application Performance Management

Issue/Introduction

We noticed that we are unable to access to the ES snapshots : 

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-platform-on-premise/23-1/Upgrade-DX-Platform/upgrade/upgrade-as-cluster-administrator/pre-upgrade-tasks-for-cluster-administrator.html#concept.dita_a25344d3-5dc1-4e05-bfd7-3779593f51ed_ESClusterSnapshotBackup

" Run the following Elasticsearch query to check the snapshots.
          GET http://ES_ROUTE/_snapshot/Repository/_all"

Expected Result:

Below is the error being reported:

" repository_exception Could not read repository data because the contents of the repository do not match its expected state. This is likely the result of either concurrently modifying the contents of the repository by a process other than this cluster or an issue with the repository's underlying storage. The repository has been disabled to prevent corrupting its contents. To re-enable it and continue using it please remove the repository from the cluster and add it again to make the cluster revoer the known state of the repository from its physical contents"

 

Environment

DX Platform 2x

Cause

Some Jarvis services were malfunctioning:

For more information regarding how to troubleshoot issues with Jarvis servies, see "AIOps - Jarvis (kafka, zookeeper, elasticSearch) Troubleshooting"

Resolution

Option 1) Restart the Jarvis services

see: AIOps - How to restart Jarvis services?

 

Option 2) Restart all DX Platform Services using <DX-HOME>/tools/dx-admin.sh stop and start :

for more information refer to : https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-platform-on-premise/23-1/Administrating/Start-or-Stop-the-Deployment.html 

Additional Information

https://knowledge.broadcom.com/external/article/190815/aiops-troubleshooting-common-issues-and.html