Namenode logs reports "Space available on volume 'null' is below threshold" and enters safe mode

Products

• Services Suite

Issue/Introduction

Symptoms:

2014-01-29 13:27:48,856 WARN org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space available on volume 'null' is 103645184, which is below the configured reserved amount 104857600
 2014-01-29 13:27:48,856 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on available disk space. Entering safe mode.
 2014-01-29 13:27:48,856 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode is ON.
 Resources are low on NN. Please add or free up more resources then turn off safe mode manually. NOTE: If you turn off safe mode before adding resources, the NN will immediately return to safe mode.

Available space on the namenode server for the volume storing namenode metadata related information

$ df -h
 Filesystem Size Used Avail Use% Mounted on
 /dev/mapper/systemvg-rootlv
 4.0G 3.7G 99M 98% /

Environment

Cause

Namenode stores it's metadata information under the path defined by parameter dfs.namenode.name.dir specified in hdfs-site.xml. Another parameter dfs.namenode.resource.du.reserved in hdfs-site.xml controls the amount of space which must be available at namenode in order for it to continue with it's operation. If the amount of space available under the directories used for namenode operations is below the threshold defined by dfs.namenode.resource.du.reserved parameter, namenode enter's safemode rendering HDFS filesystem as readonly. In such a state, even if you manually force namenode to leave safemode, it will again enter the safemode. It is a check to avoid accidental metadata corruption ensuring no data loss.

Be default, if not specified in hdfs-site.xml dfs.namenode.resource.du.reserved is 104867600 bytes (~100 MB). hdfs-site.xml can be updated with a different value if required.

Volume 'null' as indicated in the error message is misleading.

Note: Additionally, there may be other directories which can be controlled by the parameter dfs.namenode.resource.checked.volumes (if specified) and will be monitored against dfs.namenode.resource.du.reserved threshold.

Resolution

Below listed are the steps required to ensure that namenode leaves SAFE mode.

• Increase the size of the available partition / volume if there is unused space on the server or clear up unnecessary files from the directory if possible.
• Add more disk space if you are running short of space on the entire server
• Once, space concern has been addressed, run the below command to exit safemode:åÊsudo -u hdfs hdfs dfsadmin -safemode leave
• Verify if namenode has left safemode:åÊsudo -u hdfs hdfs dfsadmin -safemode get

Note: A careful planning must be done to in order to determine the space requirement of a Hadoop cluster