Memory error in App Metrics log-store vms logs and failing

Products

• Operations Manager

Issue/Introduction

Some of the common symptoms are listed below:

• Logs in the App Metrics UI are unavailable.
  
• App Metrics smoke tests are failing due to the Log Store being unavailable.
• If you curl -ks the https://metrics.<host name>.<Your domain>/integration-status, the output might be similar to:
  
  {"Log Store":false,"CF API":true,"UAA":true,"Metric Store":true,"Internal Postgres":true}
  or
  {"Log Store":true,"CF API":true,"UAA":true,"Metric Store":true,"Internal Postgres":true}

• You see "fatal error: runtime: out of memory" in the log-store-vms logs. Use this command to quickly check the logs for this error:
  
  bosh -d appMetrics-ID ssh log-store-vms -c "sudo cat /var/vcap/sys/log/log-store/log-store.stderr.log | grep -i runtime"
  
  
• You see "Cannot read corrupt tsm file" in the log-store-vms logs. Use this command to quickly check the logs for this error:
  
  bosh -d appMetrics-ID ssh log-store-vms -c "sudo cat /var/vcap/sys/log/log-store/log-store.stderr.log | grep -i 'cannot read corrupt tsm file' "
  
  
• You can see .bad extension tsm file when do "find /var/vcap/store/log-store/influxdb -name "*.bad""in log-store VMs

Environment

Product Version: 2.x

Cause

This issue is caused by the the limit of max_map_count allowed per process in the Linux VM. The limit of map count is reached because of the large retention time.

Resolution

Follow these steps to workaround this issue:

1. bosh ssh to the every log-store VM.
2. sudo -i
3. monit stop log-store
4. echo 2048000 > /proc/sys/vm/max_map_count
5. monit restart log-store