DX Platform - How to verify the Throughput for NFS Storage

Products

DX Operational Intelligence DX Application Performance Management CA App Experience Analytics

Issue/Introduction

The attached script allows to Benchmark Kubernetes persistent disk volumes with fio: Read/write IOPS, bandwidth MB/s and latency.
It is based on https://github.com/leeliu/dbench, it launches benchmark daemonset pods on all the nodes and runs the fio commands.

This script will help you verify and ensure throughput for NFS storage is as per DX Platform hardware requirement:

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-platform-on-premise/20-2/installing/software-requirements.html#concept.dita_d7ab03039be1dd1c0bbba340c2848b6d7d6349f2_HardwareRequirements

Cause

dbench-fio.sh script is part of DX Platform 21.3 release

Environment

DX Operational Intelligence 20.2
DX Application Performance Management 20.2
DX AXA 20.2
DX Platform 20.2

Resolution

1) Download and copy dbench-fio.sh to <Installer-HOME>/tools

2) Add execute permissions

cd <Installer-HOME>/tools
chmod +x dbench-fio.sh

3) If you are using Openshift, download and copy dxi-custom.scc.yml to <Installer-HOME>/tools

-Create scc: oc create -f dxi-custom.scc.yml

cd <Installer-HOME>/tools
oc create -f dxi-custom.scc.yml
securitycontextconstraints.security.openshift.io/dxi-custom created

-Apply scc to the service Account

oc adm policy add-scc-to-user dxi-custom -z dbench-pod -n <namespace>
for example
oc adm policy add-scc-to-user dxi-custom -z dbench-pod -ndxi

4) For dbench-fio options available, run: ./dbench-fio.sh --readme

5) Summary of usage

- Start test:
To start the test pass the --start parameter to the script, Example: ./dbench-fio.sh --start

- Resume test:
Test could some time upto 30-40 mins depending upon the nfs speed, If the script took longer or ssh timedout then you can resume the test
by using --resume parameter to the script, Example: ./dbench-fio.sh --resume, This will fetch the log from pods and dump into
the test summary File

- Delete test:
Using this option allows to delete failed pods, Example: ./dbench-fio.sh --delete

- Single Node test:
Allows to run dbench test only on single node in the cluster --single, for example: ./dbench-fio.sh --start --single
--resume can be used later to fetch reports

NOTE:
By default, the script identify automatically the NFS path used by the DX Platform installation
If unable to do so, it will prompt for the location the path should have folder ownership set to 1010,
Example:
cd /nfs/ca/
mkdir -vp dxi
chown -cR 1010:1010 configs

Here '/nfs/ca' is the nfs path and 'dxi' is subpath,
Provide '/nfs/ca/dxi' to the script.

6) Examples of execution:

Example#1: run script across all nodes:

./dbench-fio.sh --start

Enter namespace: dxi
Using namespace dxi
Please provide dbench-fio image dbench-fio, see Readme with --readme to this script or Press enter to use default image docker.io/karthik101/dbench-fio or Please enter 'C' to continue to exit : <press enter>
persistentvolumeclaim/dbench-pv-claim created
persistentvolume/dbench-pv created
configmap/dbench-cm created
[INFO] Launching dbench test, Please note that this operation can take several minutes, depending on the Nfs speed. **
serviceaccount/dbench-pod created
daemonset.apps/dbench-pod created
[INFO] Waiting for dbench-pod pods to run

[INFO] Waiting for dbench-pod daemonset to complete
[INFO] POD Skipping: dbench-pod-fg964
[INFO] POD Skipping: dbench-pod-m4d9t
...
[INFO] TEST completed for Node: munqa001495.bpc.broadcom.net , test_summery_2021-09-02_03-35-05.txt file for the results
[INFO] POD Skipping: dbench-pod-fg964
[INFO] POD Skipping: dbench-pod-m4d9t
...
[INFO] TEST completed for Node: munqa001498.bpc.broadcom.net , test_summery_2021-09-02_03-35-05.txt file for the results
[INFO] POD Skipping: dbench-pod-rghcv
[INFO] POD Skipping: dbench-pod-ssxxj
[INFO] POD Skipping: dbench-pod-fg964
..

[INFO] TEST completed for Node: munqa001493.bpc.broadcom.net , test_summery_2021-09-02_03-35-05.txt file for the results
ALL TESTS ON NODES COMPELTE
[INFO] test_summery_2021-09-02_03-35-05.txt file in the present directory for the test summery
Cleaning up ..

warning: --force is ignored because --grace-period is not 0.
serviceaccount "dbench-pod" deleted
daemonset.apps "dbench-pod" deleted
warning: --force is ignored because --grace-period is not 0.
persistentvolumeclaim "dbench-pv-claim" deleted
persistentvolume "dbench-pv" deleted
warning: --force is ignored because --grace-period is not 0.
configmap "dbench-cm" deleted

Example#2: Run script on a single node only:

./dbench-fio.sh --start --single

Enter namespace: dxi
Using namespace dxi

Please provide dbench-fio image dbench-fio, see Readme with --readme to this script or Press enter to use default image docker.io/karthik101/dbench-fio or Please enter 'C' to continue to exit : <press enter>
persistentvolumeclaim/dbench-pv-claim created
persistentvolume/dbench-pv created
configmap/dbench-cm created
[INFO] Launching dbench test, Please note that this operation can take several minutes, depending on the Nfs speed. **
serviceaccount/dbench-pod created
daemonset.apps/dbench-pod created
[INFO] Waiting for dbench-pod pods to run

[INFO] Waiting for dbench-pod daemonset to complete
[INFO] POD Skipping: dbench-pod-fg964
[INFO] POD Skipping: dbench-pod-m4d9t
...
[INFO] TEST completed for Node: munqa001495.bpc.broadcom.net , test_summery_2021-09-02_03-35-05.txt file for the results
[INFO] POD Skipping: dbench-pod-fg964
[INFO] POD Skipping: dbench-pod-m4d9t
...
[INFO] TEST completed for Node: munqa001498.bpc.broadcom.net , test_summery_2021-09-02_03-35-05.txt file for the results
[INFO] POD Skipping: dbench-pod-rghcv
[INFO] POD Skipping: dbench-pod-ssxxj
[INFO] POD Skipping: dbench-pod-fg964
..

Enter namespace: dxi
Using namespace dxi
Please provide dbench-fio image docker.io/karthik101/dbench-fio, see Readme with --readme to this script or Press enter to use default image docker.io/karthik101/dbench-fio or Please enter 'C' to continue to exit : <press enter>

Available Nodes:
munqa001493.bpc.broadcom.net
munqa001494.bpc.broadcom.net
munqa001495.bpc.broadcom.net
munqa001496.bpc.broadcom.net
munqa001497.bpc.broadcom.net
munqa001498.bpc.broadcom.net

Provide Node for the Dbench Test: munqa001493.bpc.broadcom.net
warning: --force is ignored because --grace-period is not 0.
serviceaccount "dbench-pod" deleted

warning: --force is ignored because --grace-period is not 0.
persistentvolumeclaim "dbench-pv-claim" deleted
persistentvolume "dbench-pv" deleted

[INFO] TEST completed for Node: munqa001493.bpc.broadcom.net , test_summery_2021-09-02_03-35-05.txt file for the results
ALL TESTS ON NODES COMPELTE
[INFO] test_summery_2021-09-02_03-35-05.txt file in the present directory for the test summery
Cleaning up ..

warning: --force is ignored because --grace-period is not 0.
serviceaccount "dbench-pod" deleted
daemonset.apps "dbench-pod" deleted
warning: --force is ignored because --grace-period is not 0.
persistentvolumeclaim "dbench-pv-claim" deleted
persistentvolume "dbench-pv" deleted
warning: --force is ignored because --grace-period is not 0.
configmap "dbench-cm" deleted

IMPORTANT:
The test could some mins and hours depending upon the NFS speed, if the script times out and exited, you can resume the test using:
./dbench-fio.sh --resume

7) Output
The script will create an output report in the present directory

Attached an example of the output : test_summery_2021-09-02_03-35-05.txt

8) Compare summary report with Recommended Throughput for NFS Storage as per documentation.

You need to check the NFS IOPS and NFS Speed

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-platform-on-premise/20-2.html