How to add a Pivotal Hadoop client node to an existing Pivotal Hadoop cluster manually

Products

• Services Suite

Issue/Introduction

Question:
How to add a Pivotal Hadoop client node to an existing Pivotal Hadoop cluster manually?

Environment

Resolution

Solution:
To add a hadoop client node to an existing cluster, you must follow the below steps

1) Install the client rpm and its dependencies packages. [Ensure that you have copied /etc/yum.repos.d/gphd.repo on the newly added client node from the cluster node, and yum is working].

Example commands:

1. yum install -y hadoop-client
2. yum install -y hive
3. yum install -y hbase
4. yum install -y oozie-client

Note: Refer to "Pivotal Hadoop Stack & Tools Reference guide" for further details for the rpm list if yum is not configured.

2) Modify core-site.xml, hdfs-site.xml under /etc/gphd/hadoop/conf directory. In this article, we will add only the relevant required parameters for secured HDFS configuration.

core-site.xml

<configuration>
 <property>
  <name>fs.defaultFS</name>
  <value>hdfs://<namenodehost_fqdn>:8020</value>
 </property>
 
 <property>
 <name>hadoop.security.authentication</name>
 <value>kerberos</value>
 </property>
 
 <property>
 <name>hadoop.security.authorization</name>
 <value>true</value>
 </property>
 </configuration>
hdfs-site.xml

</configuration>
 <property>
 <name>dfs.block.access.token.enable</name>
 <value>true</value>
 </property>
 
 <property>
 <name>dfs.client.read.shortcircuit</name>
 <value>false</value>
 </property>
 
 <property>
 <name>dfs.namenode.kerberos.principal</name>
 <value>hdfs/_HOST@REALM</value>
 </property>
 </configuration>

 

Note: If its not a secured cluster, you can copy the configuration files from any of the cluster node to the newly added node. Configuration files are stored under /etc/gphd directory on the cluster nodes for all the services.

Steps:

1) Tar the configuration file /etc/gphd on any of the cluster node

2) Remove the already existing /etc/gphd directory on the newly added client node, and untar the tar ball.

3) Ensure that the java version installed is correct. Run this commands on the client nodes being added.

ls -l `which java`
java -version
alternatives --config java
- If alternatives is not pointing to correct java version, please perform the below. Please replace jdk path according to your installation.
alternatives --install /usr/bin/java java /usr/java/jdk1.7.0_51/bin/java 3 --slave /usr/bin/javac javac /usr/java/jdk1.7.0_51/bin/java
alternative --config java
- Chose the expected java version.

4) Download and unzip the JCE (Java Cryptography Extension) file for your JDK version [For secured clusters only]

Place the local_policy.jar and US_export_policy.jar files under the directory /usr/java/default/jre/lib/security/ on the newly added client node.

5) Login as the desired user.
Note: Assuming that principal for the user has already been created.

6) Execute hadoop filesystem, mapreduce commands to verify

hdfs dfs -ls /
hadoop jar /usr/lib/gphd/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 2 2