Post-upgrade to 20.3.3/20.4, the DB grew very fast and used up space at an alarming rate. Also profiles are not being deployed/added as expected.
Other symptoms (noticed within MCS):
Issues:
Release : 20.3
Component : MON_CONFIG_SERVICE
- MCS
- mon_config_service
- UIM 20.3 and 20.4
- SSRV2AuditTrail is not being purged frequently enough
- SSRV2AuditTrail row count is ~350 million
Related parameters in the mon_config_service.cfg
garbage_cleanup_interval_minutes by default is set to 20
garbage_cleanup_batch_size is set to 10000
days_in_audit_trail_history default is 30 #as of 20.4
To keep the table smaller, set the days_in_audit_trail_history to a lesser value, e.g., 1, 3, or maybe 5 so the audit trail table does not become so large. Keep track of it each day to decide on the value you should set it to.
If you have a large number of records in the SSRV2Audttrail and SSRV2AudittrailModification tables that you want to clean up, decrease the cleanup interval minutes and decrease the batch size rather than increase the batch size.
Check and/or keep track of the row count by running the query:
select count(*) from SSRV2AuditTrail
In this case, it was originally 350M rows.
Run the stored procedure-> exec sp_spaceused in SQL Server
database size from exec sp_spaceused; was: 386454.81 MB or 386GB, unallocated ~133MB
After about 15-20 minutes or so the SSRV2AuditTrail was much smaller after truncating the tables as described above.
Run the stored procedure-> exec sp_spaceused in SQL Server to check again.
Cold start the mon_config_service probe (Deactivate-Activate).
Then check the mon_config_service.log for successes/failures.