App Usage Service startup errors and data inconsistency
Article ID: 298105
Updated On:
Products
VMware Tanzu Application Service for VMs
Issue/Introduction
This article describes a bug in older versions of App Usage Service which may cause startup errors or inconsistent usage data due to invalid events from Cloud Controller.
App Usage Service (AUS) helps track which apps and services are active in a foundation. Internally, the service relies on events reported from Cloud Controller (CC) API endpoints. Within CC, there are two sources of these events: “purge and reseed” synthetic events (the origin of this bug) and real-time events.
The purge and reseed operation is used to generate a set of synthetic start events for all running apps in a foundation. Typically, this is performed as a Support-assisted operation when recovering foundations that have experienced a data consistency issue. Unfortunately, due to a bug with how those synthetic events are generated, some events are missing critical data for AUS to properly correlate future app lifecycle events. This can have the effect of stop and restart events not being recognized which, in turn, affects usage reporting.
In addition to manual purge and reseed operations, newer versions of AUS use the operation as part of bootstrapping new installations. This behavior was introduced to better support foundations where AUS may not have been initially installed.
In some newer versions, AUS may crash when it encounters invalid, synthetic events generated by a purge and reseed operation.
Symptoms:
In some newer versions, the following error may be visible in logs from the app-usage-server app running in the system org and space:
In other affected versions, there may not be a clear indication from App Usage Service logs or monitoring. Internally, a manual reconciliation of running apps compared between Cloud Controller and AUS may confirm unexpected apps in AUS. For foundations where AUS was installed as part of the initial deployment and no other purge and reseed operation has been used, these unexpected apps should be limited to those in the system org.
App Usage Service (AUS) helps track which apps and services are active in a foundation. Internally, the service relies on events reported from Cloud Controller (CC) API endpoints. Within CC, there are two sources of these events: “purge and reseed” synthetic events (the origin of this bug) and real-time events.
The purge and reseed operation is used to generate a set of synthetic start events for all running apps in a foundation. Typically, this is performed as a Support-assisted operation when recovering foundations that have experienced a data consistency issue. Unfortunately, due to a bug with how those synthetic events are generated, some events are missing critical data for AUS to properly correlate future app lifecycle events. This can have the effect of stop and restart events not being recognized which, in turn, affects usage reporting.
In addition to manual purge and reseed operations, newer versions of AUS use the operation as part of bootstrapping new installations. This behavior was introduced to better support foundations where AUS may not have been initially installed.
In some newer versions, AUS may crash when it encounters invalid, synthetic events generated by a purge and reseed operation.
Symptoms:
In some newer versions, the following error may be visible in logs from the app-usage-server app running in the system org and space:
Mysql2::Error: Field 'app_name' doesn't have a default value: INSERT INTO `app_events` (`guid`, `occurred_at`, `process_guid`, `space_guid`, `space_name`, `org_guid`, `state`, `memory_in_mb_per_instance`, `instance_count`, `created_at`, `updated_at`) VALUES (...
In other affected versions, there may not be a clear indication from App Usage Service logs or monitoring. Internally, a manual reconciliation of running apps compared between Cloud Controller and AUS may confirm unexpected apps in AUS. For foundations where AUS was installed as part of the initial deployment and no other purge and reseed operation has been used, these unexpected apps should be limited to those in the system org.
Environment
Product Version: 2.9
Resolution
Upgrade to a supported version. The following patch versions were first to include the required bug fix.
- 2.9.6 (released 06/25/2020; release notes)
- 2.8.12 (released 06/25/2020; release notes)
- 2.7.18 (released 06/25/2020; release notes)
Feedback
Yes
No
Powered by
