Hive query fails with error "Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient"

Products

• Services Suite

Issue/Introduction

This article discusses how to fix a failed Hive query. Hive query fails with the following error:
 

"Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient"

hive> show tables;
FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

This problem results in Out Of Memory (OOM).

Environment

Cause

The Out Of Memory (OOM) error, in this case, is a result of the server not having enough memory to start HiveMetaStore service.

RCA

HiveMetaStore service is configured in hive-site.xml on the client side, but the service is not started.

For example:

/etc/gphd/hive-0.11.0_gphd_2_1_0_0/conf/hive-site.xml

<property>
<name>hive.metastore.uris
<value>thrift://hdw1.viadea.com:9083
</property>

However on hdw1:

-bash-4.1$ service hive-metastore status
hive-metastore dead but pid file exists

After observing the hive-metastore.log, you can see that reason why the service failed to start was due to an Out of Memory (OOM) error, as shown in the example below:

14/04/07 16:43:13 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
library initialization failed - unable to allocate file descriptor table - out of memory

Resolution

To resolve this issue, follow the steps below: 
1. Understand why HiveMetaStore service is not started. In this case, increase the physical memory of the server.
2. Then start the HiveMetaStore service manually using the root user.

service hive-metastore start