Enable monitoring for configuration changes with the audit probe
search cancel

Enable monitoring for configuration changes with the audit probe

book

Article ID: 128820

calendar_today

Updated On:

Products

DX Unified Infrastructure Management (Nimsoft / UIM) CA Unified Infrastructure Management On-Premise (Nimsoft / UIM) CA Unified Infrastructure Management SaaS (Nimsoft / UIM)

Issue/Introduction

Customer needs to know how to configure the audit probe to track configuration changes by users and settings

Environment

  • UIM 20.x
  • audit v9.04 or later

Cause

  • Missing documentation about the audit probe and how it works to help track user changes

Resolution

  • The audit probe does not capture all types of user interactions.
  • The audit probe has restrictions to work with IM, and it will not record who made the change if changes are made in Admin Console or MCS.
  • Furthermore, each probe has different capabilities in terms of what can be captured. Also, if you need to track user's when they make changes to probe configuration's, you have to tell it what user to track the configuration changes for when you setup audit rules/profiles.

Relevant audit tables include:

AUDIT_EVENT

AUDIT_CONFIG_CHANGE

By default, the probe will only keep 30 days of audit records in these tables. This is configurable from the probe's Configure GUI. 

Note that any changes made to a probe configuration through the monitoring tab of the UMP USM portlet, Admin Console or MCS, will not be recorded by the audit probe. 

All of the most granular information is stored in the AUDIT_EVENT table and the audit.log.


select * from audit_event

Sample rows from the audit_event table:
event_id    event_type    event_time    event_descr    origin    domain    hub    robot    probe    user_name    user_cmd    user_ip    status    config_changes    checkpoint_id

1    1    2019-03-04 12:26:31.000    Login: succeeded for audit    <hubname>    <domain_name>    <hubname>    xxxxxxxxxx    hub    NULL    login    ##.###.#.###    0    0    0
2    1    2019-03-04 12:26:50.000    Turn on audit    <hubname>    <domain_name>    <hubname>    <hubname>    controller    <username>_audit_type    ##.###.#.###    0    0    0
3    0    2019-03-04 12:26:51.000    Audit initiated    <hubname>    <domain_name>    <hubname>    <hubname>    controller    NULL    NULL    NULL    0    0    1
4    0    2019-03-04 12:26:51.000    Audit initiated    <hubname>    <domain_name>    <hubname>    <hubname>    spooler    NULL    NULL    NULL    0    0    1
5    0    2019-03-04 12:26:51.000    Audit initiated    <hubname>    <domain_name>    <hubname>    <hubname>    hdb    NULL    NULL    NULL    0    0    1
6    0    2019-03-04 12:26:51.000    Audit initiated    <hubname>    <domain_name>    <hubname>    <hubname>    cdm    NULL    NULL    NULL    0    0    1
7    0    2019-03-04 12:26:51.000    Audit initiated    <hubname>    <domain_name>    <hubname>    <hubname>    wasp    NULL    NULL    NULL    0    0    1
8    0    2019-03-04 12:26:51.000    Audit initiated    <hubname>    <domain_name>    <hubname>    <hubname>    xendesktop    NULL    NULL    NULL    0    0    1

select * from audit_event where event_id IN (select event_id from audit_config_change where new_value is NOT NULL)

event_id    event_type    event_time    event_descr    origin    domain    hub    robot    probe    user_name    user_cmd    user_ip    status    config_changes    checkpoint_id

227    1    2019-03-05 11:33:55.000    NULL    <hubname>_hub    <hubname>_domain    <hubname>_hub    <hubname>    processes    administrator    probe_config_set    ##.###.##.##    0    1    6
228    1    2019-03-05 11:43:45.000    NULL    <hubname>_hub    <hubname>_domain    <hubname>_hub    <hubname>    audit    administrator    probe_config_set    ##.###.##.##    0    1    6
229    1    2019-03-05 11:44:08.000    NULL    <hubname>_hub    <hubname>_domain    <hubname>_hub    <hubname>    processes    administrator    probe_config_set    ##.###.##.##    0    1    7
241    1    2019-03-05 11:48:50.000    NULL    <hubname>_hub    <hubname>_domain    <hubname>_hub    <hubname>    processes    audit_test    probe_config_set    ##.###.##.##    0    1    8
242    1    2019-03-05 11:49:48.000    NULL    <hubname>_hub    <hubname>_domain    <hubname>_hub    <hubname>    processes    audit_test    probe_config_set    ##.###.##.##    0    1    9
244    1    2019-03-05 11:50:09.000    NULL    <hubname>_hub    <hubname>_domain    <hubname>_hub    <hubname>    processes    audit_test    probe_config_set    ##.###.##.##    0    1    10
245    1    2019-03-05 11:51:57.000    NULL    <hubname>_hub    <hubname>_domain    <hubname>_hub    <hubname>    audit    audit_test    probe_config_set    ##.###.##.##    0    1    7
246    1    2019-03-05 11:52:31.000    NULL    <hubname>_hub    <hubname>_domain    <hubname>_hub    <hubname>    processes    audit_test    probe_config_set    ##.###.##.##    0    1    11

You can experiment with the query to achieve what you want.

When you configure the threshold, you have to pick a column and character type from the results of the query run, and make sure you set the Operator, e.g., =

Options:

1. Use the sql_response probe to run a query such as: 

select * from audit_event where... 
or 
select <attrib1,attrib2,attrib3> from audit_event where 

or something like:

select * from audit_event where event_id IN (select event_id from audit_config_change where variable = 'config_changes')

and use the Value Tab/function to alarm based on the output. 

or use the sqlserver probe and create a custom checkpoint.

Important Notes:

With hub 'Override: audit on' selected in the hub GUI and audit probe log set to loglevel 5, as soon as you make a change to a probe, e.g., the cdm probe on a robot, you should see it in the log. In this scenario, no audit key value was set at the robot level (controller):

Mar 22 13:12:04:059 [5092] audit: [0084] hub=xxxxxxx_hub robot=xxxxxxx probe=cdm config_change=yes 
Mar 22 13:12:04:059 [5092] audit: PreProcessArg - /disk/alarm/fixed/E:\, 
Mar 22 13:12:04:059 [5092] audit: PreProcessArg - active, 
Mar 22 13:12:04:059 [5092] audit: PreProcessArg - no, 
Mar 22 13:12:04:059 [5092] audit: PreProcessArg - yes, 
Mar 22 13:12:04:063 [5092] audit: [0084] change: section=/disk/alarm/fixed/E:\          key=active                no        ->yes 
Mar 22 13:12:04:063 [5092] audit: PreProcessArg - /disk/alarm/fixed/C:\, 
Mar 22 13:12:04:063 [5092] audit: PreProcessArg - active, 
Mar 22 13:12:04:063 [5092] audit: PreProcessArg - no, 
Mar 22 13:12:04:063 [5092] audit: PreProcessArg - yes, 
Mar 22 13:12:04:068 [5092] audit: [0084] change: section=/disk/alarm/fixed/C:\          key=active                no        ->yes 
Mar 22 13:12:04:068 [5092] audit: AnalyzeAuditRecord leave 
Mar 22 13:12:04:068 [5092] audit: SREPLY: status = 0(OK) ->##.##.##.##/48002 
Mar 22 13:12:04:068 [5092] audit: DataCallback leave 

As you can tell from the audit.log, it doesn't show which user made the change when you look at the audit log at loglevel 5, although the change and activity shows up in the audit probe GUI itself.

When you configure the hub to allow the robot to set the audit level, e.g., audit = 8, set in the robot (controller) probe via Raw Configure, you can see the configuration change by the user, in this case 'administrator,' in the audit.log as long as you create a NEW filter in the audit probe GUI.

Mar 22 13:18:36:011 [6124] audit: data[4]         PDS_PPDS        333 
Mar 22 13:18:36:011 [6124] audit:  event_id        PDS_I             3 84
Mar 22 13:18:36:011 [6124] audit:  event_type      PDS_I             2 1
Mar 22 13:18:36:011 [6124] audit:  event_time      PDS_I            11 1553274723
Mar 22 13:18:36:011 [6124] audit:  origin          PDS_PCH          19 xxxxxxx_hub
Mar 22 13:18:36:011 [6124] audit:  domain          PDS_PCH          22 xxxxxxx_domain
Mar 22 13:18:36:011 [6124] audit:  hub             PDS_PCH          19 xxxxxxx_hub
Mar 22 13:18:36:011 [6124] audit:  robot           PDS_PCH          15 xxxxxxx
Mar 22 13:18:36:011 [6124] audit:  probe           PDS_PCH           4 cdm
Mar 22 13:18:36:011 [6124] audit:  user_name       PDS_PCH          14 administrator
Mar 22 13:18:36:011 [6124] audit:  user_cmd        PDS_PCH          17 probe_config_set
Mar 22 13:18:36:011 [6124] audit:  user_ip         PDS_PCH          13 ##.###.##.##
Mar 22 13:18:36:011 [6124] audit:  status          PDS_I             2 0
Mar 22 13:18:36:011 [6124] audit:  config_changes  PDS_I             2 1

IMPORTANT!!! If you do NOT setup a specific filter in the audit probe GUI you will NOT see the user_name entries showing who made the change, in the audit.log

You can add a filter and test it and you will see log entries such as the ones above. Then you can delete the filter and retest it and youll notice that user_name will no longer be included in the audit log.

Here is an example of what fields were used for tracking changes by a user to the cdm probe:


 



2. It may be advantageous to use the sqlserver probe and create a custom checkpoint because you get a bit more flexibility in the QOS and alarm threshold configuration/setup. 

https://knowledge.broadcom.com/external/article?articleNumber=39422

 

Additional Information

You can use the sql_response probe to run a query on the audit tables to pull out the information and place it into an alarm.

SELECT ae.event_id, event_time, user_ip, robot,probe,USER_NAME,user_cmd,acc.operation,acc.section,acc.variable,acc.old_value,acc.new_value

  FROM [CA_UIM].[dbo].[AUDIT_EVENT] ae

  join [CA_UIM].[dbo].[AUDIT_config_change] acc on ae.event_id = acc.event_id

  where ae.probe = 'cdm'

  and ae.user_cmd = 'probe_config_set'

  and event_time > dateadd(minute, -5, getdate())

So it looks for events in the past 5 minutes and therefore the probe must run every 5 mins.

So every time you edit the cdm probe you get an alarm, then you can use that alarm to send an email.

sql_response probe profile looks like the following:

   <check_for_audit_rows>
      active = yes
      description =
      connection = ca_uim_database
      interval = 30sec
      timeout = 30sec
      query =
      file = check_for_audit_rows.sql
      query_use = file
      cursor = client
      timeout_error = major
      no_record_severity = inactive
      no_record_message = $profile/$server/$database - query returned no record
      no_record_i18n_token = as#database.sql_response.no_record
      no_record_supp_key = value
      no_record_qos =
      scheduling = rules
      source =
      <response>
         active = no
         qos = no
         network = inclusive
         type = prepare
         pings = 0
         low_severity = warning
         low_message = $profile/$server/$database - response time [$time ms] exceeded threshold [$threshold]
         low_threshold =
         low_i18n_token =
         high_severity = major
         high_message = $profile/$server/$database - response time [$time ms] exceeded threshold [$threshold]
         high_threshold =
         high_i18n_token =
         clear_severity = clear
         clear_message = $profile/$server/$database - alarm cleared: time = $time ms
         clear_i18n_token =
      </response>
      <count>
         active = no
         qos = no
         low_severity = warning
         low_message = $profile/$server/$database: $rows rows in select is $condition threshold $threshold
         low_threshold =
         low_i18n_token =
         high_severity = major
         high_message = $profile/$server/$database: $rows rows in select is $condition threshold $threshold
         high_i18n_token =
         high_threshold =
         clear_severity = clear
         clear_message = $profile/$server/$database - alarm cleared: rows = $rows
         clear_i18n_token =
         condition = >
      </count>
      <value>
         active = yes
         qos = no
         NULLs_handling = as_zero
         row_key = event_time
         <alarm_lists>
            <1>
               name = alarm_message
               columns = user_cmd
               column_type =
               low_severity = inactive
               low_message = $profile/$server/$database: value $value is $condition threshold $threshold
               low_threshold =
               high_severity = major
               high_message = $profile - $value $USER_NAME $operation, $section , $variable, $old_value, $new_value
               high_threshold = probe_config_set
               clear_severity = clear
               clear_message = $profile/$server/$database - alarm cleared, value = $value
               low_condition = =
               comparison = character
            </1>
         </alarm_lists>
      </value>
   </check_for_audit_rows>
Powered by Wolken Software