search cancel

DX OI: No data displayed in GUI with "Error fetching data"

book

Article ID: 188422

calendar_today

Updated On:

Products

DX Operational Intelligence

Issue/Introduction

No data is displayed in the DX OI GUI with "Error fetching data"


Many Pods stay Evicted with the low on resource [DiskPressure].


The "low disk watermark [85%] exceeded" message appeared in the ElastricSearch log:

[2020-01-31T06:23:34,948][INFO ][o.e.c.r.a.DiskThresholdMonitor] [IKMmjhx] low disk watermark [85%] exceeded on [IKMmjhx6Sn6YXwIgbrU15Q][IKMmjhx][/var/data/elasticsearch/nodes/0] free: 7.2gb[14.5%], replicas will not be assigned to this node

 

   

 

Cause

This problem occurs when the disk usage over 85%  on the file system which OpenShift running. 

 

Environment

OpenShift 3.x
DX Operational Intelligence: 1.3.x

Resolution

The file system should be under 85% usage rate, preferable below 30%.

Additional Information

Here is an example of expanding the CentOS7 LVM file system size.

(Initial state)
# df
Filesystem                                                  1K-blocks     Used Available Use% Mounted on
/dev/mapper/centos-root                                      52403200 42952760   9450440  82% /
(omit later)

# fdisk -l
Disk /dev/sda: 107.4 GB, 107374182400 bytes, 209715200 sectors

-- Attach the physical file system or enhance the file system size on VM. 
(This sample is enhancing 200 GB file system size on the VM settings)

# fdisk -l
Disk /dev/sda: 322.1 GB, 322122547200 bytes, 629145600 sectors

# fdisk /dev/sda
Command (m for help): <= print the partition table
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200   209715199   103808000   8e  Linux LVM

Command (m for help): <= add a new partion
Select (default p): <= primary
Partition number (3,4, default 3): <= partition #3
First sector (209715200-629145599, default 209715200):  <= enter (choose default setting)
Using default value 209715200
Last sector, +sectors or +size{K,M,G} (209715200-629145599, default 629145599):  <= enter (choose default setting)
Using default value 629145599
Partition 3 of type Linux and of size 200 GiB is set

Command (m for help): <= change a partition's system id
Partition number (1-3, default 3): 3
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'

Command (m for help): <= print the partition table
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200   209715199   103808000   8e  Linux LVM
/dev/sda3       209715200   629145599   209715200   8e  Linux LVM

Command (m for help): <= write table to disk and exit
The partition table has been altered!

-- Now reboot the Linux system.

# pvcreate /dev/sda3
  Physical volume "/dev/sda3" successfully created.

# vgextend centos /dev/sda3
  Volume group "centos" successfully extended

# lvextend -L +204800 /dev/mapper/centos-root
  Size of logical volume centos/root changed from 50.00 GiB (12800 extents) to 250.00 GiB (64000 extents).
  Logical volume centos/root successfully resized.

# resize2fs -f /dev/mapper/centos-root

# xfs_growfs /dev/mapper/centos-root
data blocks changed from 13107200 to 65536000

# df -h
Filesystem                                                   Size  Used Avail Use% Mounted on
/dev/mapper/centos-root                                      250G   42G  209G  17% /
(omit later)

Attachments