Expanding capture and index storage on Security Analytics

Products

• Security Analytics

Issue/Introduction

New storage space can be added to capture more packets and extend the depth of the indexes.

Environment

The instructions and scripts have been tested in Security Analytics 8.2.8.

Resolution

The following will require root access from the command line.  This will need to be done during a maintenance window due to the fact that all capture will need to be stopped and many Security Analytics services will need to be shut down.

First record the output of the following commands for configuration reference:

• df -h
• lsblk --output 'NAME,KNAME,MAJ:MIN,WWN,MODEL,SIZE,SERIAL'
• multipath -ll


Initial Configuration

1. Install the Storage Array and configure the IP Address, admin username, and password referencing the Security Analytics Gen8 Hardware Installation and Configuration Guide.
2. Follow the procedures through the end of page 30. Your server is already configured so page 31 and part of 32 can be skipped.
3. Begin again at “Configure Network Settings for Storage Arrays” on page 32 and continue through the end of page 33.

Shutting Down Capturing Interfaces and the services

1. List the capturing devices with dscapture --status  Look for any interfaces with the label “CAPTURING”
2. Stop all capturing ports with dscapture --stop ethX  where "X" Is the number of the interface
3. Check again that there are no capturing ports with dscapture --status
4. Stop Security Analytics services with scotus stop

Configuring the new storage expansion hardware

NOTE:  Do not run the powervault_clear.py command.  This will delete all capture and index data.  The process for expanding storage will allow you to add storage without losing any existing capture and index data.

1. Continue on page 35 and run the powervault_probe command against the IP address of the new storage array.
2. Run the powervault_config script on page 36 pointing to the json file just created from the probe command.
3. Reboot the appliance entering:  reboot
4. Once back up, stop the Solera services again.  You may need to run it a second time to get all the services stopped successfully:  scotus stop
5. Find the new storage logical devices with the following command:  lsblk --output 'NAME,KNAME,MAJ:MIN,WWN,MODEL,SIZE,SERIAL'

The existing storage volumes will have the following group names:  

captureVG-captureLV
indexVG-indexLV

These devices can be considered as existing capture and existing index.  The new volumes on the new storage will not have any group names listed, but they should have similar volume sizes as the existing logical volumes.  Double check that the new dm-<num>'s are going to be associated with the correct group-name. The new logical device names are often sequential to the old, but this is not guaranteed.

Once the new 'dm' volume devices have been identified, it is time to expand the existing storage to include the new storage.

Expand the Storage

NOTE:  adding the -n (--noExec) parameter will show commands but not execute anything.  Running expand-ds-storage --help will display:

Usage: expand-ds-storage -ci [-nravDh] drives
        [-c, --capture]            expand the capture storage
        [-i, --index  ]            expand the index storage
        [-n, --noExec ]            show commands but do not execute
        [-r, --reboot ]            automatically reboot
        [-a, --allow  ]            allow mismatch stripe member sizes
        [-v, --verbose]            verbose (show commands)
        [-D, --debug  ]            debugging output, show even more commands
        [-h, --help   ]            show this usage

NOTE:  These dm- names are only for reference.  Those on your server will be different

1. Run the following command to expand the capture volumes:  expand-ds-storage -c /dev/dm-5 /dev/dm-6 /dev/dm-7 /dev/dm-8
2. When the script is done, enter N to postpone rebooting.
3. Run the following command to expand the index volumes: expand-ds-storage -i /dev/dm-9
4. When the script is done, enter N to postpone rebooting.
5. Verify the new logical volumes are properly included in the logical group name by running the lsblk command:  
           lsblk --output 'NAME,KNAME,MAJ:MIN,WWN,MODEL,SIZE,SERIAL'
6. Reboot the server:  reboot

After reboot, run df -h and verify that both capture (/pfs) and index (/var/lib/solera/meta) has increased in size.

Restart capture on the previous ethX ports that were capturing before:  dscapture --start ethX (where X is your port number)