MapReduce jobs fails with "Not able to initialize user directories in any of the configured local directories"

Products

• Services Suite

Issue/Introduction

Symptoms:

Executing MapReduce job fails with the following error message:

14/02/27 01:59:06 INFO mapreduce.Job: Job job_1393464189937_0030 running in uber mode : false
14/02/27 01:59:06 INFO mapreduce.Job: map 0% reduce 0%
14/02/27 01:59:06 INFO mapreduce.Job: Job job_1393464189937_0030 failed with state FAILED due to: Application application_1393464189937_0030 failed 1 times due to AM Container for appattempt_1393464189937_0030_000001 exited with exitCode: -1000 due to: java.io.IOException: Not able to initialize user directories in any of the configured local directories for user hdfs
 at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.createUserLocalDirs(DefaultContainerExecutor.java:357)
 at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:88)
 at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:860)

Environment

Resolution

• Identify the yarn directories specified using the parameter yarn.nodemanager.local-dirs from /etc/gphd/hadoop/conf/yarn-site.xml on yarn node manager.
• Delete the files / folders under usercache from all the directories listed in yarn-site.xml and all the node managers.

Example: rm -rf /data/dn/yarn/nm-local-dir/usercache/*