Map reduce job failed with "Could not find any valid local directory for output/attempt_xxxx_xxxx_m_x/file.out"

Products

• Services Suite

Issue/Introduction

Symptoms:

Map Reduce job failed with the following error message: 

Error: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/attempt_1426272186088_184484_m_000232_4/file.out

	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:398)

	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)

	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)

	at org.apache.hadoop.mapred.YarnOutputFiles.getOutputFileForWrite(YarnOutputFiles.java:84)

	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1820)

	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489)

	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)

Environment

Cause

Map Reduce task during execution will store intermediate data onto local directories, these directories are specified by the parameter "mapreduce.cluster.local.dir" (Old deprecated name: mapred.local.dir) specified in the mapred-site.xml.

During job processing, the map reduce framework looks for the directories specified by mapreduce.cluster.local.dir parameter and verifies if there is enough space on the directories listed to create the intermediate file. If there is no directory which has the required space, the map reduce job will fail with the error as shown above.

Resolution

To resolve this issue, follow the suggestions below: 

1. Ensure that there is enough space on to the local directories based on the requireme amount of data that needs to be processed.

2. You may compress the intermediate output files to minimize the space consumption.