PXF Service can't start - "ERROR: Can't find PXF instance, maybe call init?"

Products

• Services Suite

Issue/Introduction

When attempting to startup PXF service via Ambari, it fails on one server host with the error message below:

resource_management.core.exceptions.Fail: Execution of 'service pxf-service restart' returned 1. ERROR: can't find PXF instance, maybe call init?
ERROR: cant find PXF instance, maybe call init?

There is insufficient memory resource on the failed host to be able to start up the Java VM for PXF service.

Environment

OS: RHEL 6.x

Resolution

RCA 

PXF service script will perform "callcatalina.sh version" to check if the environment is ready or not. If it fails, it will report "ERROR: cant find PXF instance, maybe call init?"

$instance_root/$instance_name/bin/catalina.sh version > /dev/null 2>&1

In this case "catalina.sh version" fails to run due to insufficient memory resource and no host to start the Java VM, as shown below.

[root@XXX bin]# ./catalina.sh version
Using CATALINA_BASE: /var/pxf/pxf-service
Using CATALINA_HOME: /var/pxf/pxf-service
Using CATALINA_TMPDIR: /var/pxf/pxf-service/temp
Using JRE_HOME: /usr
Using CLASSPATH: /var/pxf/pxf-service/bin/bootstrap.jar:/var/pxf/pxf-service/bin/tomcat-juli.jar
Using CATALINA_PID: /var/run/pxf/catalina.pid
Error occurred during initialization of VM
Could not reserve enough space for object heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
[root@XXX bin]# sudo -u pxf ./catalina.sh version
Using CATALINA_BASE: /var/pxf/pxf-service
Using CATALINA_HOME: /var/pxf/pxf-service
Using CATALINA_TMPDIR: /var/pxf/pxf-service/temp
Using JRE_HOME: /usr
Using CLASSPATH: /var/pxf/pxf-service/bin/bootstrap.jar:/var/pxf/pxf-service/bin/tomcat-juli.jar
Using CATALINA_PID: /var/run/pxf/catalina.pid
Error occurred during initialization of VM
Could not reserve enough space for object heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Further check found that several stale Java processes were hogging too much memory:

758974 scdf 20 0 35.6g 3.0g 11m S 0.0 1.2 0:39.88 java
72967 scdf 20 0 33.8g 407m 10m S 0.3 0.2 0:09.28 java
35953 mapred 20 0 1585m 406m 22m S 0.7 0.2 1:13.41 java
42951 hcat 20 0 2279m 362m 29m S 0.0 0.1 0:18.05 java

 

Workaround

To resolve this issue, kill the processes consuming a lot of memory to free more memory resource and start the PXF service again.