App Metrics UI shows "log store not available" in App Metrics 2.0.0 - 2.0.5

Products

• Operations Manager

Issue/Introduction

In App Metrics versions 2.0.0 - 2.0.5, the following symptoms are observed:

• App Metrics UI shows "log store not available".
• App logs missing from App Metrics.
• On the affected log store VMs, monit summary shows all processes running correctly.

If you try to open a raw connection to port 8080 to the log store VMs, some fail while others succeed:

$ nc <log-store-vm1-IP> 8080 -v
nc: connect to <log-store-vm1-IP> port 8080 (tcp) failed: Connection refused
$ nc <log-store-vm2-IP> 8080 -v
Connection to <log-store-vm2-IP> 8080 port [tcp/http-alt] succeeded!
$ nc <log-store-vm3-IP> 8080 -v
nc: connect to <log-store-vm3-IP> port 8080 (tcp) failed: Connection refused

In a log store VM with a failed connection, you see the following errors:

log-store/log-store.stderr.log:{"level":"error","timestamp":"2021-01-26T21:42:43.033950915Z","caller":"tsm1/file_store.go:544","message":"Cannot read corrupt tsm file, renaming","engine":"tsm1","service":"filestore","path":"/var/vcap/store/log-store/influxdb/38/data/logs/default/1608595200000000000/000000905-000000001.tsm","id":746,"error":"cannot allocate memory"}

 

Root cause

"When the db engine detects a corrupt tsm data file upon db startup procedure, the re-naming of that file fails due to a file lock on the data file created by the db engine itself." For more information, refer to Renaming corrupt data files fail.

Environment

Product Version: 2.1

Resolution

Workaround

Re-pave App Metrics with a brand new installation.

Fix

We have added a newer version of InfluxDB to log-store in App Metrics 2.0.6, which includes a fix. For more information, refer to Bump Influxdb: 1.8.4. 

Note: Prior to moving to App Metrics 2.0.6, the tile needs to be repaved.