apmservices-nass pod restarting : org.rocksdb.RocksDBException: Sst file size mismatch: ./data/nass_spooldb

Products

• DX Operational Observability

Issue/Introduction

apm services-nass pod is restarting so performance metrics and inventory information is not accessible

from apmservices-nass-pod log:

Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'schedulers': Unsatisfied dependency expressed through field 'metricStore'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'metricStore': Unsatisfied dependency expressed through field 'spoolManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'spoolManager': Unsatisfied dependency expressed through field 'familyManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'familyManager': Invocation of init method failed; nested exception is com.ca.apm.common.db.DatabaseException: java.io.IOException: org.rocksdb.RocksDBException: Sst file size mismatch: ./data/nass_spooldb/4240410.sst. Size recorded in manifest 83239, actual size 1080

Environment

DX O2

Cause

NAS database is corrupted.

There are many cases of how it can happen, but mostly it is related to NFS connection issues and use of the old NFS version (v3). Version 4.1 is recommended as one that is much more resilient as per documentation

Resolution

1. Scale down apmservices-nass-001 deployment

2. Identify DX manager pod and connect to the pod:

Example in openshift: oc rsh apmservices-manager-001-7588688454-9cqqd

3. *IMPORTANT* Backup nass spool for analysis:

cd /data.all/apmservices/nass-001/data/

tar cvzf spooldbbackup.tar.gz nass_spooldb/

4. Cleanup spool db:
rm nass_spooldb/*

5. Scale up apmservices-nass-001 deployment

If you have any doubt or need assistance contact Broadcom Support for assistance