YARN is giving an alert that the NodeManager is unhealthy and the local-dirs are bad

Products

• Services Suite

Issue/Introduction

Symptoms:
The following alert appears in Ambari for a YARN NodeManager:

Error Message:

NodeManager Health Summary

1 NodeManager is unhealthy.

 NodeManager Health

1/1 local-dirs are bad: /data/hadoop/yarn/local; 1/1 log-dirs are bad: /data/hadoop/yarn/log

Environment

Cause

The most likely cause of local-dirs are bad errors is the available disk space on the node exceeding yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage(default value is 90%).

Resolution

Check space utilization on the node and the disk where you are getting the unhealthy node errors and compare it to the value for yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage on Ambari. For example:

[root@nodeN ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 9.8G 7.1G 2.3G 76% /
tmpfs 3.8G 0 3.8G 0% /dev/shm
/dev/sda1 477M 77M 375M 17% /boot
/dev/sda7 55G 52G 3G 95% /data
/dev/sda2 20G 45M 19G 1% /home
/dev/sda3 9.8G 24M 9.2G 1% /tmp

If the utilization in the yarn disk (in this case /data) is above the limit set by yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage,try these options:

• Free up some disk space
• Increase the value for yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage through Ambari.

Restart the NodeManager services to pick up the changes.