Expanding the disk space capacity of a Collector node of a VCF Operations for Networks deployment

Products

VCF Operations for Networks

Issue/Introduction

NOTE: VCF Operations for Networks was formerly named Aria Operations for Networks (AON), and prior to that was named vRealize Network Insight (vRNI).

Environment

VCF Operations for Networks

Cause

When a VCF Operations for Networks Collector is deployed from the .ova downloaded from the Broadcom Support Portal, it contains one hard disk of 250G in size.

The disk capacity can be expanded using the instructions at Increase the Disk Size

Resolution

GENERAL PRINCIPLES:

Editing the hard disk size to be a larger size than the one in the .ova template, after deploying from the downloaded .ova (but before the first power on) will not allow VCF Operations for Networks to take advantage of any increase in size. Any addition to the size will simply be wasted as it will subtract from the available space in the datastore.
The proper way to increase the disk size, per the documentation at Increase the Disk Size, is to add an additional hard disk, after the deployed Collector has been fully configured and integrated into the VCF Operations for Networks deployment, with the Collector VM in a powered on state.
The size of an additional hard disk is constrained only by the available datastore space when you add the new hard disk.
The limit to the number of hard disks that can be added to a Collector is determined by Configuration Maximums for the number of hard disks per virtual machine (VM)
Only one additional hard disk can be added at a time. After an additional hard disk is added, the Collector node has a cron job that runs every 5 minutes, to detect if a new hard disk has been added.
The detection of the additional hard disk will be documented in the file /home/ubuntu/logs/disk_util.log which can be interrogated using a command such as "sudo cat /home/ubuntu/logs/disk_util.log"
- Output will be of this format:
  - ####-##-##T##:##:##.030Z: Found new disk
    ####-##-##T##:##:##.060Z: Creating PV: /dev/sdc
    ####-##-##T##:##:##.169Z: Extending VG: vg
    ####-##-##T##:##:##.228Z: Extending LV: /dev/vg/var Size: ##.#G
    ####-##-##T##:##:##.570Z: DONE
To see the disk space allocation by Mount Point inside the VM, log into the Collector node using the support user and enter the df -h command.
- Output will be of this format (below would be for a fresh deployment):
  - Filesystem Size Used Avail Use% Mounted on
    tmpfs 1.2G 980K 1.2G 1% /run
    /dev/mapper/vg-root 29G 6.4G 21G 24% /
    tmpfs 5.9G 8.0K 5.9G 1% /dev/shm
    tmpfs 5.0M 0 5.0M 0% /run/lock
    /dev/sda1 2.0G 157M 1.7G 9% /boot
    none 5.9G 0 5.9G 0% /run/shm
    /dev/mapper/vg-tmp 25G 628K 24G 1% /tmp
    /dev/mapper/vg-home 30G 1.5G 27G 6% /home
    /dev/mapper/vg-var 49G 356M 46G 1% /var
    /dev/mapper/vg-var+log 49G 881M 45G 2% /var/log
    /dev/mapper/vg-var+log+audit 20G 18M 19G 1% /var/log/audit
    tmpfs 1.2G 0 1.2G 0% /run/user/1000
    tmpfs 1.2G 0 1.2G 0% /run/user/1001
To see the disk space allocation for the Collector, log into the VCF Operations for Networks GUI, select Settings --> Infrastructure and Updates and scroll down to the Collector VM(s) section, and click on the blue Collector under the Name column that you wish to check. Then, in the Overview section, under Key Properties, observe the numbers to the right of the Total Disk (GB) and Total Data Disk (GB) lines. Below is an example screen shot of what is shown for a newly deployed Collector node:

NOTES:

- After adding an additional hard disk, it may take up to 30 minutes after the disk is recognized by the cron job, to reflect the new space allocation in the GUI.
- However, the additional space allocation change can be seen in the /dev/mapper/vg-var Filesystem using the df -h command per the above example, immediately after the disk is recognized as documented in the /home/ubuntu/logs/disk_util.log
- After the first additional hard disk is added, the amount of disk space mentioned in the "Extending LV: /dev/vg/var Size: " event in the /home/ubuntu/logs/disk_util.log will be just slightly smaller than the size of the additional hard disk added.
- However, the incremental increase in Total Disk (GB) and Total Data Disk (GB) lines as a result of adding the first additional hard disk will be approximately 39.5G more than the size of the additional hard disk added. That increase (total of the size of the additional hard disk + 39.5G) will be reflected in the output of the "df-h" command for the Filesyetem /dev/mapper/vg-var
- After adding a second additional hard disk (or third, or any after that), the incremental increase in Total Disk (GB) and Total Data Disk (GB) lines will be approximately the size of the additional hard disk(s) added.
- The amount of disk space actually allocated to the Collector VM can be seen in vCenter and/or ESXi and will be shown in the same way as for any other virtual machine (VM).

Additional Information

See also KB Resolving disk space issues with an Aria Operations for Networks Platform or Collector Node for discussion of "Disk utilization is at 76%." alerts.