Alerts show Ambari is unable to contact HiveServer2 on port 10000

Products

• Services Suite

Issue/Introduction

Symptoms:

Alerts and Health Check in the HiveServer show the HiveServer2 process: "Connection to <hiveserver hostname> on port 10000 failed":


Note: HiveServer2 works correctly. The alert in Ambari shown above is misleading.

Environment

Cause

HiveServer2 has two transport modes:

• binary - this is the default mode and by default will listen on port 10000
• HTTP - This is the non-default mode and by default will listen on port 10001

The ports used may change depending on what has been configured in Ambari. This is what the configuration looks like by default:

If hive.server2.transport.mode is set to "binary" then HiveServer2 will listen on the port set in Hive Server Port, in this case 10000 (this is the default setting).

If hive.server2.transport.mode is set to "http" then HiveServer2 will listen on the port set in hive.server2.thrift.http, in this case 10001.

Ambari is not aware of the possible port changes and  the transport mode Ambari will only check for HiveServer2 on the port 10000 which causes this reporting issue if the HTTP mode is used.

Resolution

This is a reporting issue, if HiveServer2 is working correctly, the alert can be ignored. However, if HiveServer2 goes down, the Hive administrator may not be correctly notified.
 

To fully resolve this reporting issue:

1. Upgrade to Ambari 2.1.2
2. Make sure Ambari Metrics is installed
3. Restart HIVE