Ambari Web UI is slow or unresponsive

Products

• Services Suite

Issue/Introduction

Symptoms:
After running out of disk space on one node in a HD cluster, the Ambari Web UI can become very slow or unresponsive. Even after the disk space issue has been resolved.

Environment

Cause

The Ambari server process was busy contacting Ambari metrics collector and degraded the performance.
 

RCA 

1. As shown in /var/log/ambari-server/ambari-server.log, the Ambari server is constantly connecting to the Ambari metric collector but is failing:

[root@AYCAPSU01AS203 ambari-server]# tail -f ambari-server.log
03 Apr 2017 11:36:31,906 ERROR [qtp-client-9752] MetricsRequestHelper:87 - Error getting timeline metrics. Can not connect to collector, socket error.
03 Apr 2017 11:36:35,891  WARN [qtp-ambari-agent-9809] HeartBeatHandler:222 - Old responseId received - response was lost - returning cached response
03 Apr 2017 11:36:40,968  WARN [qtp-ambari-agent-9787] HeartBeatHandler:222 - Old responseId received - response was lost - returning cached response
03 Apr 2017 11:36:48,105 ERROR [qtp-client-9750] MetricsRequestHelper:87 - Error getting timeline metrics. Can not connect to collector, socket error.
03 Apr 2017 11:37:04,295 ERROR [qtp-client-9755] MetricsRequestHelper:87 - Error getting timeline metrics. Can not connect to collector, socket error.
03 Apr 2017 11:37:11,348  WARN [qtp-ambari-agent-9811] HeartBeatHandler:222 - Old responseId received - response was lost - returning cached response
03 Apr 2017 11:37:20,487 ERROR [qtp-client-9755] MetricsRequestHelper:87 - Error getting timeline metrics. Can not connect to collector, socket error. 

2. The Ambari metrics collector is being hosted on the node where the disk space issue had occurred.

3. As a result, the Java process of the Ambari metric collector is not running in a good state to a disk space issue and failed to process a connection request from the Ambari server.

Resolution

Restart the Ambari metrics service through the Ambari Web UI. 
 

If the Ambari Web UI is not responsive, restart Ambari server first and quickly logon to the Web UI. Then restart the Ambari metrics service immediately.