Cloud Controller Database size grows rapidly after upgrading Splunk Tile to version 1.4.2
Article ID: 432112
Updated On:
Products
Issue/Introduction
After upgrading Splunk Tile to version 1.4.2 the Cloud Controller Database ( CCDB ) size increases significantly by tens of gigabytes. Specifically the events table as seen with below query.
to run these queries first ssh into any EAR internal msyql vm and launch the mysql client as root.
mysql --defaults-file=/var/vcap/jobs/pxc-mysql/config/mylogin.cnf ccdb
Query that shows table sizes
mysql> SELECT table_name, ROUND (data_length / 1024 / 1024, 2) AS data_mb, ROUND (index_length / 1024 / 1024, 2) AS index_mb, ROUND (data_length + index_length / 1024 / 1024, 2) AS total_mb FROM information_schema.tables WHERE table_schema = 'ccdb' ORDER BY total_mb DESC;
+-------------------------------------------+---------+----------+-----------+
| TABLE_NAME | data_mb | index_mb | total_mb |
+-------------------------------------------+---------+----------+-----------+
| events |24662.02 | 38896.08 | 63558.09 |
| schema_migrations | 0.08 | 0.00 | 81920.00 |
| apps | 0.05 | 0.14 | 49152.14 |
| revisions | 0.05 | 0.06 | 49152.06 |
| app_usage_events | 0.05 | 0.05 | 49152.05 |
| droplets | 0.02 | 0.16 | 16384.16 |
| service_bindings | 0.02 | 0.14 | 16384.14 |
| deployments | 0.02 | 0.13 | 16384.13 |
| tasks | 0.02 | 0.13 | 16384.13 |
| buildpack_lifecycle_data | 0.02 | 0.11 | 16384.11 |
| route_mappings | 0.02 | 0.11 | 16384.11 |
| routes | 0.02 | 0.11 | 16384.11 |
| service_instances | 0.02 | 0.11 | 16384.11 |
Upon further inspection you will see most events are generated by actor "splunk-nozzle"
SELECT e.type, e.actor, e.actor_type, COUNT(*) AS count FROM events e WHERE e.created_at > NOW() - INTERVAL 1 HOUR GROUP BY e.type, e.actor, e.actor_type ORDER BY count DESC LIMIT 20;
+--------------------------------------+--------------------------------------+------------+-------+
| type | actor | actor_type | count |
+--------------------------------------+--------------------------------------+------------+-------+
| audit.app.environment_variables.show | splunk-nozzle | user | 725028|
| audit.user.organization_user_add | #################################### | user | 14 |
| audit.user.organization_user_add | #################################### | user | 10 |
Cause
When USE_LABELS_FOR_SPLUNK_INDEX is enabled Splunk will query the cloud controller api for each app to fetch environmental variables info. A recent commit introduced this behavior by a fix for Splunk indexing
For each api call splunk makes there will be a corresponding audit event recorded by cloud controller in the ccdb.events table.
Resolution
A community issue was opened with Splunk here. Recommend following up with Splunk Support and referencing this issue if you have any questions.
If you encounter this issue and require an immediate workaround then we recommend disabling the splunk nozzle and rolling back the tile version to previous until you can migrate to version 1.4.2. Alternatively if you wish to stay on 1.4.2 you can disable "USE_ENV_VAR_FOR_SPLUNK_INDEX" and enable "USE_LABELS_FOR_SPLUNK_INDEX", which will inform the splunk nozzle to use app labels instead of env vars for indexing.
See Splunk install docs regarding environment variable settings
https://cloudfoundry-community.github.io/splunk-firehose-nozzle/develop/environment-variables/
Feedback
