Hive Cli fails to start with the error message "TezSession has already shutdown"

Products

• Services Suite

Issue/Introduction

Symptoms:

The Hive CLI fails to start with the following error message:

[hdfs@datanode2 hive]$ hive
WARNING: Use "yarn jar" to launch YARN applications.

Logging initialized using configuration in file:/etc/hive/2.3.2.0-2950/0/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1467175461700_0015 failed 2 times due to AM Container for appattempt_1467175461700_0015_000002 exited with exitCode: 1
For more detailed output, check application tracking page:http://datatnode2.local.lab:8088/cluster/app/application_1467175461700_0015Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_e12_1467175461700_0015_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 

Review the YARN application log by running yarn log -applicationId application_1467175461700_0015 and observe the following:

2016-06-29 15:27:05,647 [INFO] [main] |launcher.ContainerLauncherImpl|: Upper limit on the thread pool size is 500
2016-06-29 15:27:05,647 [INFO] [main] |history.HistoryEventHandler|: Initializing HistoryEventHandler withrecoveryEnabled=true, hist
oryServiceClassName=org.apache.tez.dag.history.logging.ats.ATSV15HistoryLoggingService
2016-06-29 15:27:05,649 [INFO] [main] |service.AbstractService|: Service org.apache.tez.dag.history.HistoryEventHandler failed in st
ate INITED; cause: org.apache.tez.dag.api.TezReflectionException: Unable to load class: org.apache.tez.dag.history.logging.ats.ATSV1
5HistoryLoggingService
org.apache.tez.dag.api.TezReflectionException: Unable to load class: org.apache.tez.dag.history.logging.ats.ATSV15HistoryLoggingServ
ice
 at org.apache.tez.common.ReflectionUtils.getClazz(ReflectionUtils.java:46)
 at org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:78)
 at org.apache.tez.dag.history.HistoryEventHandler.serviceInit(HistoryEventHandler.java:61)
 at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
 at org.apache.tez.dag.app.DAGAppMaster.initServices(DAGAppMaster.java:1671)
 at org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:554)

Environment

Cause

This is caused by a known Ambari software bug 15041.

Resolution

Follow these steps to resolve the issue: 

1. Open Ambari.
2. Go to TEZ / Configs / Advanced tez-site.
3. Locate the configuration tez.history.logging.service.class.
4. Replace the value org.apache.tez.dag.history.logging.ats.ATSV15HistoryLoggingService with the new value: org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService"
5. Save the configuration changes.
6. Restart all services that Ambari is asking you to restart.