Troubleshoot app usage service on VMWare Tanzu Application Service (TAS) for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

This article demonstrates general troubleshooting steps when hitting an issue with App Usage Service.

App Usage Service on VMWare Tanzu Application Service (TAS) for VMs is composed of three apps, all ruby, under system Org and system Space:

• app-usage-server: API endpoint for getting data
• app-usage-worker: Worker that collects and aggregates data
• app-usage-scheduler: Scheduler for data collection

App-usage-scheduler triggers the following jobs:

• Load app usage data from ccdb and import into app_usage_service database every 5 minutes.
• Load service usage data from ccdb and import into app_usage_service database every 5 minutes.
• Update monthly usage summary at 15:00 UTC every day.
• Some other jobs to delete or roll over very older data.

App-usage-worker executes those jobs app-usage-scheduler triggers.

App-usage-server serves RESTful requests from:

• Apps Manager > accountingReport
• curl /system_report/app_usages, /system_report/service_usage, etc. Please refer to Reporting App, Task, and Service Instance Usage for more details. 

Resolution

1. App-usage-server is the API endpoint.
   1. First check if it's reporting any errors in the logs. All the apps are under system Org, system Space.
   2. Run cf target -o system -s system, then cf logs app-usage-server --recent to view app recent logs.
2. App-usage-server reads data from app_usage_service database. If app-usage-server itself functions correctly but no data or incorrect data is returned, check if there are any errors in app-usage-scheduler and app-usage-worker logs.
3. App-usage-worker loads data from ccdb database / app_usage_events, service_usage_events tables, and inserts this data into app_usage_service database, app_events / service_events tables. Check if there is valid data in those tables. Follow the below steps for accessing ccdb and app_usage_service when using internal database for Tanzu Application Service:
   1. $ bosh -d DEPLOYMENT ssh mysql/0
   2. $ sudo -i
   3. # mysql --defaults-file=/var/vcap/jobs/pxc-mysql/config/mylogin.cnf
   4. mysql> use app_usage_service; or mysql> use ccdb;
4. Other jobs and tables in the  app_usage_service database are executed or created based on data in app_events / service_events tables. If the data is not available in the two tables for any reasons, RESTful API will return an unexpected response. 
5. If the problem can not be resolved with the above steps, collect the logs of all three apps and open a support ticket.