Map Reduce jobs on a secured cluster fails to create app directories

Products

• Services Suite

Issue/Introduction

Symptoms:

After securing a cluster, if the permission on the cluster directories are not setup properly, the Map Reduce jobs fail with an error message similar to the one below:

main : user is gpadmin
 Can't open / in /data1/yarn/nm-local-dir/usercache/gpadmin/appcache/application_1397684926758_0002 - Permission denied
 Can't open / in /data2/yarn/nm-local-dir/usercache/gpadmin/appcache/application_1397684926758_0002 - Permission denied
 Can't open / in /data3/yarn/nm-local-dir/usercache/gpadmin/appcache/application_1397684926758_0002 - Permission denied
 Can't open / in /data4/yarn/nm-local-dir/usercache/gpadmin/appcache/application_1397684926758_0002 - Permission denied
 Can't open / in /data5/yarn/nm-local-dir/usercache/gpadmin/appcache/application_1397684926758_0002 - Permission denied
 Did not create any app directories

Environment

Cause

The above error message indicates that the application failed to open "/" directory while trying to write application logs on the local filesystem.

Depending on the directory on which MapReduce jobs fail to read, you can verify the permission and ensure that all the users have read and write permissions.

In this particular case, "/" directory did not have the appropriate permissions. After changing it to 755, the MapReduce jobs will be able to create the directories successfully.

gpadmin:Fullrack@mdw $ ls -tlrd /
drwx--x--x 41 root root 4096 Apr 10 19:19 /

Container executor program ensures that the path specified by yarn.nodemanager.local-dirs in yarn-site.xml must have at-least readonly permission on all of the parent directories starting from "/".

A snippet from container-executor.c indicates to check the "/" directory. This snippet extends to check for all the child directories as well.

int mkdirs(const char* path, mode_t perm) {
  char *buffer = strdup(path);
  char *token;
  int cwd = open("/", O_RDONLY);
  if (cwd == -1) {
    fprintf(LOGFILE, "Can't open / in %s - %s\n", path, strerror(errno));
    free(buffer);
    return -1;
  }

Resolution

Provide read and write privileges to the directory displayed in the error message as shown below: 

gpadmin:Fullrack@mdw $ ls -tlrd /
drwxr-xr-x 41 root root 4096 Apr 10 21:19 /