Nodemanager fails to start with container-executor java.io.IOException: error=13, Permission denied

Products

• Services Suite

Issue/Introduction

Yarn nodemanager may fail to start in a secure cluster with the an error message: container-executor: java.io.IOException: error=13, Permission denied

The following error message is from the yarn-nodemanager logs.

2014-03-28 20:52:22,266 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: Security is enabled on NodeManager. Creating ContainerTokenSecretManager
2014-03-28 20:52:22,732 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: setsid exited with exit code 0
2014-03-28 20:52:22,736 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
org.apache.hadoop.yarn.YarnException: Failed to initialize container executor
 at org.apache.hadoop.yarn.server.nodemanager.NodeManager.init(NodeManager.java:144)
 at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:321)
 at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:359)
Caused by: java.io.IOException: Cannot run program "/usr/lib/gphd/hadoop-yarn/bin/container-executor": java.io.IOException: error=13, Permission denied
 at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
 ... 2 more
Caused by: java.io.IOException: java.io.IOException: error=13, Permission denied
 at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
 ... 7 more
2014-03-28 21:14:27,520 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:

container-executor.cfg is configured with the below permissions, where yarn is the group name to which this file is associated. Thus, yarn user must be associated with yarn group so as to execute the container-executor program, else it will bail out due to permission issues.

[root@phd11-nn ~]# ls -tlr /usr/lib/gphd/hadoop-yarn/bin/container-executor
---Sr-s--- 1 root yarn 34200 Nov 12 18:53 /usr/lib/gphd/hadoop-yarn/bin/container-executor

In this case, yarn user was not associated with yarn group, but both were available.

root@tstr400059:$ id yarn 
uid=99523(yarn) gid=5910(hadoop) groups=5910(hadoop)

Resolution

Associate the yarn user with the yarn group available in the system and try starting the node manager again:

root@tstr400059:$ usermod -G yarn yarn
root@tstr400059:$ id yarn 
uid=99523(yarn) gid=5910(hadoop) groups=5910(hadoop),497(yarn) 
root@tstr400059:$ service hadoop-yarn-nodemanager start
starting nodemanager, logging to /var/log/gphd/hadoop-yarn/yarn-yarn-nodemanager-tstr400059.out
[ OK ]