Namenode failed while loading fsimage with GC overhead limit exceeded

Products

• Services Suite

Issue/Introduction

Symptoms:

During startup, NameNode fails to load fsimage into memory.

2014-05-14 17:36:56,806 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file /data/hadoop/nn/dfs/name/current/fsimage_0000000000252211550 using no compression
2014-05-14 17:36:56,806 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 29486731
2014-05-14 17:54:40,401 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.lang.OutOfMemoryError: GC overhead limit exceeded
 at java.util.Arrays.copyOf(Arrays.java:2271)
 at java.util.zip.ZipCoder.getBytes(ZipCoder.java:89)

Environment

Cause

Every file, directory and block in HDFS (Hadoop Distributed File System) is stored as an object in the NameNode and occupies around 150 bytes of memory. The value 150 is not a fixed number, but is generally used as a rule of thumb. In this particular instance there is around:

• 29+ million files, of which around 25+ million files were generated during NameNode stress benchmarking test (nnbench).
• The size of fsimage file was 2.7 GB. This translates to an average of 90-100kb of memory occupied in NameNode memory per file.
• Heap Size is by default 1GB.

Due to a lower Heap Size value and a higher amount of fsimage size to be loaded in memory, the NameNode Garbage Collector process is spending too much time to reclaim memory causing GC overhead limit errors.

Resolution

Follow the steps below to resolve this issue: 

1. Identify an approximate value for Xmx or Xms parameters using the following formula:

[root@hdm1 current]# sudo -u hdfs hdfs oiv -p XML -printToScreen -i fsimage_0000000000252211550 -o /tmp/a | egrep "BLOCK_ID|INODE_ID" | wc -l | awk '{printf "Objects=%d : Suggested Xms=%0dm Xmx=%0dm\n", $1, (($1 / 1000000 )*1024), (($1 / 1000000 )*1024)}'

Objects=29000000 : Suggested Xms=29696m Xmx=29696m

2. Edit HADOOP_NAMENODE_OPTS parameter in /etc/gphd/hadoop/conf/hadoop-env.sh on NameNode to the suggested Xmx and Xms values.

3. Start the NameNode:

service hadoop-hdfs-namenode start