vIDM appliance has no space left on device /db for audit data

Products

VMware Aria Suite

Issue/Introduction

This article provides steps to add space in the vIDM appliance adding config to limit audit data.

vIDM appliance /db file system is almost full, for example usage exceeds about 90% and more.
You will see these errors:

check /db file system usage by df command as follows
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/db_vg-db 10076 8318 1246 87% /db

Getting vIDM log-bundle fails with the error: No space left on device.

Environment

VMware Identity Manager 3.3.x

Cause

vIDM appliance uses device /db for Elasticsearch, RabbitMQ and internal DB PostgreSQL.
/db
|- data
|- elasticsearch
|- log
|- lost+found
|- temp
A large number of audit data for Elasticsearch uses almost all space on device /db file system.
As the default settings vIDM appliance keeps audit data forever, the vIDM file system will eventually fill up.

Resolution

To resolve this issue, add a policy to limit audit data in the /usr/local/horizon/conf/runtime-config.properties file:

Log in to the vIDM appliance.
Edit the /usr/local/horizon/conf/runtime-config.properties file.

vi /usr/local/horizon/conf/runtime-config.properties

Add these two rows to the file:

analytics.deleteOldData=true -----> The default value is false.
analytics.maxQueryDays=90 -----> The default value is 90, which indicates the number of days to keep the audit date. It is recommended to be 90 days to get 12-week statistics.

Reboot the vIDM appliance to reflect the new setting.
Remove the audit data manually that has already expired for more than 90 days using the curl command for cleanup. You can use wildcards to delete the audit data:

Examples 1:
curl -XDELETE http://localhost:9200/v3_2016*
Explanation: Deletes everything from 2016.

Examples 2:
curl -XDELETE http://localhost:9200/v3_2016-12*
Explanation: Deletes everything from December 2016.

Backup: If the curl command fails, run the following commands:

rabbitmqctl list_queues | grep analytics.null
Explanation: Lists the RabbitMQ queue

rabbitmqctl purge_queue -.analytics.null
Explanation: Purges the queue
For vIDM versions 3.3.4 and higher, Workspace ONE Access 20.10 and higher, you will need to do the following workaround, as the command above will fail on the new Photon releases.

wget 'https://raw.githubusercontent.com/rabbitmq/rabbitmq-management/v3.7.20/bin/rabbitmqadmin'
chmod +x rabbitmqadmin
sed -i 's|#!/usr/bin/env python|#!/usr/bin/env python3|' rabbitmqadmin
mv rabbitmqadmin /usr/sbin/
rabbitmqadmin -q list queues | grep analytics
rabbitmqadmin purge queue name=-.analytics.127.0.0.1

Note:

* in the file name is a wildcard.
Do not use the rm commands to remove the audit data unless all nodes are stopped.

Additional Information

There is also a related KB on increasing the size of the /db file system.
- How to Increase vIDM appliance disk space
The rm command is not suitable for use with a distributed, replicated data store, so removing the files when Elasticsearch/Opensearch is running will cause problems.
It is recommended to use the “curl” command to perform the removal safely.