EM failed to start. org.rocksdb.RocksDBException: IO error: lock data/metadata/LOCK: Resource temporarily unavailable

book

Article ID: 126467

calendar_today

Updated On:

Products

CA Application Performance Management Agent (APM / Wily / Introscope) INTROSCOPE

Issue/Introduction

EM failed to start. org.rocksdb.RocksDBException: IO error: lock data/metadata/LOCK: Resource temporarily unavailable


 

Cause

The issue is due transaction log corruption due to unclean shutdown of the Introscope EM (kill, lack of space, EM crash).

This error indicated that the metadata for the SmartStor database on this EM has either corrupted or being locked for some other reasons (for eg. concurrent access), and is preventing the EM to start up successfully.
 

Environment

Applies to any APM 10.5.0, 10.5.1

Resolution

To address the file corruption root cause, please do the following:

1. Back up the current <EM_home>\data\metadata directory
2. Delete everything inside the <EM_Home>\data\metadata directory
3. Start the EM

Deleting the metadata db on the EM should have no significant impact, as the main agents SmartStor data are stored on the collectors instead.

Note: It may be due to concurrent access of the metadata files by other existing EM process, and rebooting the server has seemed to released the lock