AIOps - How to verify throughput for NFS (IOPs and Speed)

Products

• DX Operational Intelligence
• DX Application Performance Management
• CA App Experience Analytics

Issue/Introduction

The attached script allows to Benchmark Kubernetes persistent disk volumes with fio: Read/write IOPS, bandwidth MB/s and latency.
It is based on https://github.com/leeliu/dbench, it launches benchmark daemonset pods on all the nodes and runs the fio commands.

This script will help you verify and ensure throughput for NFS storage is as per DX Platform hardware requirement: 

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-platform-on-premise/20-2/installing/Hardware-software-requirements.html#concept.dita_d7ab03039be1dd1c0bbba340c2848b6d7d6349f2_HardwareRequirements

 

 

Cause

dbench-fio.sh script is part of DX Platform 21.3 release

Environment

DX Operational Intelligence 20.2.x
DX Application Performance Management 20.2.x
DX AXA 20.2.x
DX Platform 20.2.x

Resolution

1. Download and extract attached files to <Installer-HOME>/tools

1630561213902__dxi-custom.scc.tar
1633446491626__dbench-fio.tar

2. Make the dxi-custom.scc.sh executable:

    chmod +x dbench-fio.sh

3. If you are using Openshift, apply below steps:

a) Create scc: oc create -f dxi-custom.scc.yml

cd <Installer-HOME>/tools
oc create -f dxi-custom.scc.yml

b) Apply scc to the service Account

oc adm policy add-scc-to-user dxi-custom -z dbench-pod -n <namespace>

for example:
oc adm policy add-scc-to-user dxi-custom -z dbench-pod -ndxi

4. To access to the README, execute :

./dbench-fio.sh --readme 

5. A quick summary of commands:

START TEST:                    ./dbench-fio.sh --start
DELETE TEST PODS:      ./dbench-fio.sh --delete
RESUME AFTER DELAY: ./dbench-fio.sh --resume
TEST SINGLE NODE:      ./dbench-fio.sh --node <Test Name>
SHOW HELP:                   ./dbench-fio.sh --help
SHOW README:             ./dbench-fio.sh --readme

    Start test 
    Allows to run all the tests on all nodes in the cluster, use --start parameter to the script,

    Example: ./dbench-fio.sh --start
   
    Resume test:
    If the script took longer or ssh timedout then you can resume the test by using --resume parameter to the script,

    Example: ./dbench-fio.sh --resume

    This will fetch the log from pods and dump into the test summery File
   
    Delete test:
    Using this option allows to delete failed pods,

    Example: ./dbench-fio.sh --delete
   

    Single Node test:
    Allows to run dbench test only on single node 

    Option a) run individual tests like: randread, randwrite, read_latency, write_latency, read_seq, write_seq, randrw
   
    Example: ./dbench-fio.sh --node randread

    Option b) run all the above tests on single node

    Example: ./dbench-fio.sh --node all 

    If the script took longer or ssh timedout then you can resume the test by using --resume parameter to the script,
   

6. Once script execution is completed, compare the output with the recommended values as per documentation: https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-platform-on-premise/20-2.html

a) Compare the results with columns 3 and 5 : NFS IOPs and NFS Speed
b) If NFS server is running from one of he kuberentes / openshift severs, then compare the results with columns 2 and 4: Local IOPs and Local Speed

Additional Information

DX AIOPs - Troubleshooting, Common Issues and Best Practices