DX OI: No data displayed in GUI with "Error fetching data"
book
Article ID: 188422
calendar_today
Updated On:
Products
DX Operational Intelligence
Issue/Introduction
No data is displayed in the DX OI GUI with "Error fetching data"
Many Pods stay Evicted with the low on resource [DiskPressure].
The "low disk watermark [85%] exceeded" message appeared in the ElastricSearch log:
[2020-01-31T06:23:34,948][INFO ][o.e.c.r.a.DiskThresholdMonitor] [IKMmjhx] low disk watermark [85%] exceeded on [IKMmjhx6Sn6YXwIgbrU15Q][IKMmjhx][/var/data/elasticsearch/nodes/0] free: 7.2gb[14.5%], replicas will not be assigned to this node
Environment
OpenShift 3.x DX Operational Intelligence: 1.3.x
Cause
This problem occurs when the disk usage over 85% on the file system which OpenShift running.
Resolution
The file system should be under 85% usage rate, preferable below 30%.
Additional Information
Here is an example of expanding the CentOS7 LVM file system size.
(Initial state) # df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/centos-root 52403200 42952760 9450440 82% / (omit later)
# fdisk /dev/sda Command (m for help): p <= print the partition table Device Boot Start End Blocks Id System /dev/sda1 * 2048 2099199 1048576 83 Linux /dev/sda2 2099200 209715199 103808000 8e Linux LVM
Command (m for help): n <= add a new partion Select (default p): p <= primary Partition number (3,4, default 3): 3 <= partition #3 First sector (209715200-629145599, default 209715200): <= enter (choose default setting) Using default value 209715200 Last sector, +sectors or +size{K,M,G} (209715200-629145599, default 629145599): <= enter (choose default setting) Using default value 629145599 Partition 3 of type Linux and of size 200 GiB is set
Command (m for help): t <= change a partition's system id Partition number (1-3, default 3): 3 Hex code (type L to list all codes): 8e Changed type of partition 'Linux' to 'Linux LVM'
Command (m for help): p <= print the partition table Device Boot Start End Blocks Id System /dev/sda1 * 2048 2099199 1048576 83 Linux /dev/sda2 2099200 209715199 103808000 8e Linux LVM /dev/sda3 209715200 629145599 209715200 8e Linux LVM
Command (m for help): w <= write table to disk and exit The partition table has been altered!