Logs not available on app metrics dashboard for some apps

Products

• VMware Tanzu Application Service

Issue/Introduction

Logs for some apps are not available on App Metrics UI. App metrics UI show Error : Log Store error 


  

Environment

App Metrics 2.2.2

Cause

This issue is caused by the the limit of max_map_count allowed per process in the Linux VM. The limit of map count is reached because of the large retention time.
Some of the common symptoms are listed below:

• Logs for some app in the App Metrics UI are unavailable.Out put of https://metrics.SYS.DOMAIN.COM/integration-status, will not show all true:

{"Log Store":true,"CF API":true,"UAA":true,"Metric Store":true,"Internal Postgres":true}

• You see "Cannot read corrupt tsm file" in the log-store-vms logs. Use this command to quickly check the logs for this error:

bosh -d appMetrics-ID ssh log-store-vms -c "sudo cat /var/vcap/sys/log/log-store/log-store.stderr.log | grep -i 'cannot read corrupt tsm file'"

• You can see .bad extension tsm file when do find /var/vcap/store/log-store/influxdb -name "*.bad" in log-store VMs

Resolution

1. bosh ssh to the every log-store VM.
2. sudo -i
3. monit stop log-store
4. echo 2048000 > /proc/sys/vm/max_map_count 
5. monit restart log-store