Ambari user interface is extremely slow

Products

• Services Suite

Issue/Introduction

Symptoms:

The Ambari Web user interface is extremely slow when checking service performance metrics and the ambari-server.log shows the Ganglia host or server is not live or known.
 

Error messages from the ambari-server.log:

:
04:42:59,509 INFO [qtp334484437-166379] GangliaPropertyProvider:452 - Ganglia host is not live
04:43:00,693 INFO [qtp334484437-166379] GangliaPropertyProvider:452 - Ganglia host is not live
04:43:05,500 INFO [qtp334484437-166379] GangliaPropertyProvider:452 - Ganglia host is not live
04:43:06,905 INFO [qtp334484437-166379] GangliaPropertyProvider:452 - Ganglia host is not live
04:43:11,503 INFO [qtp334484437-166379] GangliaPropertyProvider:452 - Ganglia host is not live
04:43:12,394 WARN [qtp334484437-166379] GangliaReportPropertyProvider:116 - Attempting to get metrics but the Ganglia server is unknown. Resource=Resource : Cluster
Properties:
{Clusters={cluster_name=phdcluster, version=PHD-3.0}} : Cluster=phdcluster
04:43:12,396 WARN [qtp334484437-166379] GangliaReportPropertyProvider:116 - Attempting to get metrics but the Ganglia server is unknown. Resource=Resource : Cluster
Properties:
{Clusters={cluster_name=phdcluster, version=PHD-3.0}} : Cluster=phdcluster
:

Environment

Cause

With Ambari versions before v2.1.2, Apache HBase per region metrics are enabled by default. When a cluster has too many regions, it creates a severe performance issue.

The example cluster shown above has 2600 regions and each region had 30 metrics. This means there are 78,000 (2600*30) metrics to be maintained. This causes the performance issue.

Resolution

Follow the steps below to solve this issue: 

1. Locate the two files below under /var/lib/ambari-server/resources/stacks/PHD/2.0.6/services/HBASE/package/templates.

hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2
hadoop-metrics2-hbase.properties-GANGLIA-RS.j2

2. Add the two entries below to each file.

*.source.filter.class=org.apache.hadoop.metrics2.filter.GlobFilter
hbase.*.source.filter.exclude=*Regions*

3. Do a rolling restart of Hbase region servers.