DX OI: No data displayed in GUI with "Error fetching data"
Article ID: 188422
Updated On:
Products
DX Operational Intelligence
Issue/Introduction
No data is displayed in the DX OI GUI with "Error fetching data"
Many Pods stay Evicted with the low on resource [DiskPressure].
Many Pods stay Evicted with the low on resource [DiskPressure].
The "low disk watermark [85%] exceeded" message appeared in the ElastricSearch log:
| [2020-01-31T06:23:34,948][INFO ][o.e.c.r.a.DiskThresholdMonitor] [IKMmjhx] low disk watermark [85%] exceeded on [IKMmjhx6Sn6YXwIgbrU15Q][IKMmjhx][/var/data/elasticsearch/nodes/0] free: 7.2gb[14.5%], replicas will not be assigned to this node |
|
Environment
OpenShift 3.x
DX Operational Intelligence: 1.3.x
DX Operational Intelligence: 1.3.x
Cause
This problem occurs when the disk usage over 85% on the file system which OpenShift running.
Resolution
The file system should be under 85% usage rate, preferable below 30%.
Additional Information
Here is an example of expanding the CentOS7 LVM file system size.
(Initial state)
# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/centos-root 52403200 42952760 9450440 82% /
(omit later)
# fdisk -l
Disk /dev/sda: 107.4 GB, 107374182400 bytes, 209715200 sectors
-- Attach the physical file system or enhance the file system size on VM.
(This sample is enhancing 200 GB file system size on the VM settings)
# fdisk -l
Disk /dev/sda: 322.1 GB, 322122547200 bytes, 629145600 sectors
# fdisk /dev/sda
Command (m for help): p <= print the partition table
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 2099199 1048576 83 Linux
/dev/sda2 2099200 209715199 103808000 8e Linux LVM
Command (m for help): n <= add a new partion
Select (default p): p <= primary
Partition number (3,4, default 3): 3 <= partition #3
First sector (209715200-629145599, default 209715200): <= enter (choose default setting)
Using default value 209715200
Last sector, +sectors or +size{K,M,G} (209715200-629145599, default 629145599): <= enter (choose default setting)
Using default value 629145599
Partition 3 of type Linux and of size 200 GiB is set
Command (m for help): t <= change a partition's system id
Partition number (1-3, default 3): 3
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'
Command (m for help): p <= print the partition table
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 2099199 1048576 83 Linux
/dev/sda2 2099200 209715199 103808000 8e Linux LVM
/dev/sda3 209715200 629145599 209715200 8e Linux LVM
Command (m for help): w <= write table to disk and exit
The partition table has been altered!
-- Now reboot the Linux system.
# pvcreate /dev/sda3
Physical volume "/dev/sda3" successfully created.
# vgextend centos /dev/sda3
Volume group "centos" successfully extended
# lvextend -L +204800 /dev/mapper/centos-root
Size of logical volume centos/root changed from 50.00 GiB (12800 extents) to 250.00 GiB (64000 extents).
Logical volume centos/root successfully resized.
# resize2fs -f /dev/mapper/centos-root
# xfs_growfs /dev/mapper/centos-root
data blocks changed from 13107200 to 65536000
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 250G 42G 209G 17% /
(omit later)
(Initial state)
# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/centos-root 52403200 42952760 9450440 82% /
(omit later)
# fdisk -l
Disk /dev/sda: 107.4 GB, 107374182400 bytes, 209715200 sectors
-- Attach the physical file system or enhance the file system size on VM.
(This sample is enhancing 200 GB file system size on the VM settings)
# fdisk -l
Disk /dev/sda: 322.1 GB, 322122547200 bytes, 629145600 sectors
# fdisk /dev/sda
Command (m for help): p <= print the partition table
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 2099199 1048576 83 Linux
/dev/sda2 2099200 209715199 103808000 8e Linux LVM
Command (m for help): n <= add a new partion
Select (default p): p <= primary
Partition number (3,4, default 3): 3 <= partition #3
First sector (209715200-629145599, default 209715200): <= enter (choose default setting)
Using default value 209715200
Last sector, +sectors or +size{K,M,G} (209715200-629145599, default 629145599): <= enter (choose default setting)
Using default value 629145599
Partition 3 of type Linux and of size 200 GiB is set
Command (m for help): t <= change a partition's system id
Partition number (1-3, default 3): 3
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'
Command (m for help): p <= print the partition table
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 2099199 1048576 83 Linux
/dev/sda2 2099200 209715199 103808000 8e Linux LVM
/dev/sda3 209715200 629145599 209715200 8e Linux LVM
Command (m for help): w <= write table to disk and exit
The partition table has been altered!
-- Now reboot the Linux system.
# pvcreate /dev/sda3
Physical volume "/dev/sda3" successfully created.
# vgextend centos /dev/sda3
Volume group "centos" successfully extended
# lvextend -L +204800 /dev/mapper/centos-root
Size of logical volume centos/root changed from 50.00 GiB (12800 extents) to 250.00 GiB (64000 extents).
Logical volume centos/root successfully resized.
# resize2fs -f /dev/mapper/centos-root
# xfs_growfs /dev/mapper/centos-root
data blocks changed from 13107200 to 65536000
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 250G 42G 209G 17% /
(omit later)
Attachments
Feedback
Yes
No
Powered by
