Friday, 31 July 2015

Hadoop 2.6.0 Single Node Setup on Fedora

Dear viewers,

This post will help you to single node setup of hadoop 2.6.0 on Fedora or similar systems.

Steps:

1. Install Java if not exists. (Version 1.6 +)
use sudo yum install java-package-name
After installation, cross check it by below commands

[pavan@Pavan ~]$ java -version
java version "1.7.0_b147-icedtea"
OpenJDK Runtime Environment (fedora-2.1.fc17.6-x86_64)
OpenJDK 64-Bit Server VM (build 22.0-b10, mixed mode)

[pavan@Pavan ~]$ which java
/usr/bin/java

[pavan@Pavan ~]$ whereis java
java: /bin/java /usr/bin/java /etc/java /lib64/java /usr/lib64/java /usr/share/java /usr/share/man/man1/java.1.gz




2. set up ssh certificate
[pavan@Pavan ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/pavan/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/pavan/.ssh/id_rsa.
Your public key has been saved in /home/pavan/.ssh/id_rsa.pub.
The key fingerprint is:
51:79:b2:85:c8:20:28:fd:3c:b6:77:23:35:86:94:3c pavan@Pavan
The key's randomart image is:
+--[ RSA 2048]----+
| . ....+ ..o     |
|. o  .E o.+ o    |
| . o . o.  =     |
|    = . +..      |
|   . o oS.       |
|    . o o        |
|     . o .       |
|                 |
|                 |
+-----------------+
[pavan@Pavan ~]$ cat ~/.ssh/id_rsa >> ~/.ssh/authorized_keys
[pavan@Pavan ~]$ ssh localhost
pavan@localhost's password:

3. Download Hadoop-2.6.0 and extract it
You may download it from  http://hadoop.apache.org/releases.html
Extract it and put it in /usr/local directory. (You can decide destination directory of your choice).
In my case I extracted it at /usr/local directory

[pavan@Pavan ~]$ ls /usr/local/
bin  games   hadoop-2.6.0  lib    libexec  share
etc  include lib64  sbin     src

4. Set up envorionment variable
Go to home directory and open .bashrc file in text editor. If you can not see .bashrc file in home directory, then press ctrl+h, you will see it.
Add following lines to .bashrc

[pavan@Pavan ~]$ vi .bashrc

#Hadoop variables
# Modify JAVA_HOME and HADOOP_INSTALL as per your requirements

export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.3.x86_64
export HADOOP_INSTALL=/usr/local/hadoop-2.6.0
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
#end of Hadoop variable declaration

To apply changes to .bashrc run following command

[pavan@Pavan ~]$ source .bashrc


5. Open /usr/local/hadoop-2.6.0/etc/hadoop/hadoop-env.sh in text editor find below line

export JAVA_HOME=${JAVA_HOME}
and replace it with
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.3.x86_64

6. Check hadoop version
You shall get similar to this

[pavan@Pavan ~]$ hadoop version
Hadoop 2.6.0
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1
Compiled by jenkins on 2014-11-13T21:10Z
Compiled with protoc 2.5.0
From source with checksum 18e43357c8f927c0695f1e9522859d6a
This command was run using /usr/local/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0.jar

7. Configure xml files hadoop-2.6.0
Four files at directory /usr/local/hadoop-2.6.0/etc/hadoop need to be modified.
A. core-site.xml
B. mapred-site.xml
C. yarn-site.xml
D. hdfs-site.xml

A. Open file core-site.xml and paste following under <configuration> </configuration>

<property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
</property>
<property>
    <name>hadoop.tmp.dir</name>   
    <value>/usr/local/hadoop-2.6.0/tmp</value>
</property>

B. Open file mapred-site.xml and paste following under <configuration> </configuration>

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

C. Open file yarn-site.xml and paste following under <configuration> </configuration>
<property>
            <name>yarn.resourcemanager.resource-tracker.address</name>
            <value>localhost:9025</value>
</property>

D. Set up default directories and for namenode and datanode and then edit hdfs-site.xml fileNote that these directories must be in the users home directory, incase of fedora, otherwise the combined space in HDFS doesn't show full access to provided storage to the system.

[pavan@Pavan ~]$ cd ~
[pavan@Pavan ~]$ pwd
/home/pavan
[pavan@Pavan ~]$ mkdir -p hadoopdata/hdfs/namenode
[pavan@Pavan ~]$ mkdir -p hadoopdata/hdfs/datanode
[pavan@Pavan ~]$ vi /usr/local/hadoop-2.6.0/etc/hadoop/hdfs-site.xml

# add following in <configuration> </configuration
#Default replication value is 3, but you can set it as per your choice.

<property>
        <name>dfs.replication</name>
        <value>1</value>
</property>


8. Format namenode
# As this is fresh installation, formatting name node will not harm your data, because it do not contains any data. Formatting namenode will erase hadoop data. So just be sure when you format hdfs next time.

[pavan@Pavan ~]$ hdfs namenode -format
15/07/31 17:39:42 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = Pavan/192.168.5.239
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.6.0
STARTUP_MSG:   classpath = /usr/local/hadoop-2.6.0/etc/hadoop:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/hadoop-annotations-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/jsr305-1.3.9.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/activation-1.1.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/gson-2.2.4.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/hadoop-auth-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/htrace-core-3.0.4.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/curator-framework-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/commons-el-1.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/asm-3.2.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/commons-collections-3.2.1.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/junit-4.11.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/curator-client-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/xz-1.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/hadoop-nfs-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0-tests.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/htrace-core-3.0.4.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-el-1.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/hadoop-hdfs-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/hadoop-hdfs-2.6.0-tests.jar:/usr/local/hadoop-2.6.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/jline-0.9.94.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/activation-1.1.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/asm-3.2.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/guice-3.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/xz-1.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-api-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-registry-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-common-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-client-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/xz-1.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-tests.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.0.jar:/usr/local/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.6.0.jar:/contrib/capacity-scheduler/*.jar
STARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1; compiled by 'jenkins' on 2014-11-13T21:10Z
STARTUP_MSG:   java = 1.7.0_b147-icedtea
************************************************************/
15/07/31 17:39:42 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
15/07/31 17:39:42 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-baa43796-9698-4563-8c45-c3c53740104d
15/07/31 17:39:44 INFO namenode.FSNamesystem: No KeyProvider found.
15/07/31 17:39:44 INFO namenode.FSNamesystem: fsLock is fair:true
15/07/31 17:39:44 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
15/07/31 17:39:44 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
15/07/31 17:39:44 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
15/07/31 17:39:44 INFO blockmanagement.BlockManager: The block deletion will start around 2015 Jul 31 17:39:44
15/07/31 17:39:44 INFO util.GSet: Computing capacity for map BlocksMap
15/07/31 17:39:44 INFO util.GSet: VM type       = 64-bit
15/07/31 17:39:44 INFO util.GSet: 2.0% max memory 888.9 MB = 17.8 MB
15/07/31 17:39:44 INFO util.GSet: capacity      = 2^21 = 2097152 entries
15/07/31 17:39:44 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
15/07/31 17:39:44 INFO blockmanagement.BlockManager: defaultReplication         = 1
15/07/31 17:39:44 INFO blockmanagement.BlockManager: maxReplication             = 512
15/07/31 17:39:44 INFO blockmanagement.BlockManager: minReplication             = 1
15/07/31 17:39:44 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
15/07/31 17:39:44 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks  = false
15/07/31 17:39:44 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
15/07/31 17:39:44 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
15/07/31 17:39:44 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
15/07/31 17:39:44 INFO namenode.FSNamesystem: fsOwner             = pavan (auth:SIMPLE)
15/07/31 17:39:44 INFO namenode.FSNamesystem: supergroup          = supergroup
15/07/31 17:39:44 INFO namenode.FSNamesystem: isPermissionEnabled = true
15/07/31 17:39:44 INFO namenode.FSNamesystem: HA Enabled: false
15/07/31 17:39:44 INFO namenode.FSNamesystem: Append Enabled: true
15/07/31 17:39:44 INFO util.GSet: Computing capacity for map INodeMap
15/07/31 17:39:44 INFO util.GSet: VM type       = 64-bit
15/07/31 17:39:44 INFO util.GSet: 1.0% max memory 888.9 MB = 8.9 MB
15/07/31 17:39:44 INFO util.GSet: capacity      = 2^20 = 1048576 entries
15/07/31 17:39:44 INFO namenode.NameNode: Caching file names occuring more than 10 times
15/07/31 17:39:44 INFO util.GSet: Computing capacity for map cachedBlocks
15/07/31 17:39:44 INFO util.GSet: VM type       = 64-bit
15/07/31 17:39:44 INFO util.GSet: 0.25% max memory 888.9 MB = 2.2 MB
15/07/31 17:39:44 INFO util.GSet: capacity      = 2^18 = 262144 entries
15/07/31 17:39:44 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
15/07/31 17:39:44 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
15/07/31 17:39:44 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
15/07/31 17:39:44 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
15/07/31 17:39:44 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
15/07/31 17:39:44 INFO util.GSet: Computing capacity for map NameNodeRetryCache
15/07/31 17:39:44 INFO util.GSet: VM type       = 64-bit
15/07/31 17:39:44 INFO util.GSet: 0.029999999329447746% max memory 888.9 MB = 273.1 KB
15/07/31 17:39:44 INFO util.GSet: capacity      = 2^15 = 32768 entries
15/07/31 17:39:44 INFO namenode.NNConf: ACLs enabled? false
15/07/31 17:39:44 INFO namenode.NNConf: XAttrs enabled? true
15/07/31 17:39:44 INFO namenode.NNConf: Maximum size of an xattr: 16384
15/07/31 17:39:44 INFO namenode.FSImage: Allocated new BlockPoolId: BP-837338973-192.168.5.239-1438344584385
15/07/31 17:39:44 INFO common.Storage: Storage directory /usr/local/hadoop-2.6.0/data/dfs/name has been successfully formatted.
15/07/31 17:39:44 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
15/07/31 17:39:44 INFO util.ExitUtil: Exiting with status 0
15/07/31 17:39:44 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Pavan/192.168.5.239
************************************************************/


9. Start hadoop service.
Either you can start individual service or you can start all. These daemons are available at /usr/local/hadoop-2.6.0/sbin

[pavan@Pavan ~]$ cd /usr/local/hadoop-2.6.0/sbin/
[pavan@Pavan sbin]$ start-all.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [localhost]
pavan@localhost's password:
localhost: starting namenode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-pavan-namenode-Pavan.out
pavan@localhost's password:
localhost: starting datanode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-pavan-datanode-Pavan.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
RSA key fingerprint is fb:23:08:a5:a8:4c:97:23:82:b5:49:08:e8:ad:ee:b4.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.
pavan@0.0.0.0's password:
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop-2.6.0/logs/hadoop-pavan-secondarynamenode-Pavan.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop-2.6.0/logs/yarn-pavan-resourcemanager-Pavan.out
pavan@localhost's password:
localhost: starting nodemanager, logging to /usr/local/hadoop-2.6.0/logs/yarn-pavan-nodemanager-Pavan.out



10. Hadoop configuration is done. To check the java process status run following

[pavan@Pavan sbin]$ jps
7347 DataNode
7654 ResourceManager
7947 NodeManager
8044 Jps
7508 SecondaryNameNode
7220 NameNode


11. To see Web UI of hadoop overview, put following in browser
localhost:50070





12. To stop all processes run following[pavan@Pavan sbin]$ stop-all.sh

Congratulation !!! you have successfully setup hadoop 2.6.0 single node cluster.

Thanks.


References: 
1. http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/
2. https://accretioninfinity.wordpress.com/2013/06/11/installing-hadoop-for-fedora-oracle-linuxsingle-node-cluster/

9 comments:

Anonymous said...

piyush@piyush-Inspiron-N4050 /usr/local/hadoop-2.6.0/sbin $ jps
12723 SecondaryNameNode
12888 ResourceManager
12553 DataNode
13342 Jps
13182 NodeManager

NameNode doesn't start on start-all.sh. Is it necessary?

Pavan Jaiswal said...

Dear Piyush,

This could be a common problem not to see namenode process running. Try following:

1. piyush@piyush-Inspiron-N4050 /usr/local/hadoop-2.6.0/sbin$ stop-all.sh
2. piyush@piyush-Inspiron-N4050 /usr/local/hadoop-2.6.0/sbin$ start-all.sh
3. piyush@piyush-Inspiron-N4050 /usr/local/hadoop-2.6.0/sbin$ jps

If its still not working then try following:
1. piyush@piyush-Inspiron-N4050 /usr/local/hadoop-2.6.0/sbin$ stop-all-sh
2. piyush@piyush-Inspiron-N4050 /usr/local/hadoop-2.6.0/sbin$ hdfs namenode -format
3. piyush@piyush-Inspiron-N4050 /usr/local/hadoop-2.6.0/sbin$ start-all.sh
4. piyush@piyush-Inspiron-N4050 /usr/local/hadoop-2.6.0/sbin$ jps

Hope this will help you. You may acknowledge your jps status.

Thanks.

Ashwin said...

[ashwin@localhost ~]$ jps
25377 NodeManager
25043 ResourceManager
24403 NameNode
24843 SecondaryNameNode
27224 Jps

It is not showing DataNode. I have done stop-all.sh and after that start-all.sh, still DataNode is absent.

Pavan Jaiswal said...

Dear Ashwin,

Follow below process:
1. $stop-all.sh
2. You are suppose to replace clusterid value of namenode version file (which you will see in hadoop data directory. In my case path is /usr/local/hadoop-2.6.0/tmp/dfs/name/current/version) with the clusterid value of datanode version file, (path is /usr/local/hadoop-2.6.0/tmp/dfs/data/current/version). Save the file after modification and close it.
3. $start-all.sh
4. $jps

If you still not able to see the datanode process running then, try following
1. $stop-all.sh
2. Delete the "current" directory available in data directory. (in my case path is /usr/local/hadoop-2.6.0/tmp/dfs/name/current/)
3. $hadoop namenode -format
4. $start-all.sh
5. $jps

Hope this time, datanode will start.

You may acknowledge the status.

Unknown said...

There are lots of information about latest technology, like Hadoop cluster is a special type of computational cluster designed specifically for storing and analyzing huge amounts of unstructured data in a distributed computing environment. This information seems to be more unique and interesting. Thanks for sharing.

Big Data Training Chennai | Big Data Course in Chennai | Big Data Hadoop Training in Chennai

Unknown said...

Hadoop is a free,Java-based programming framework that supports the processing of large data sets in a distributed computing environment.Hadoop course is more useful for IT professionals.
Thanks,
Hadoop Training in Chennai | Bigdata Training in Chennai

Unknown said...

Thanks for sharing this article.. You may also refer http://www.s4techno.com/blog/2016/07/11/hadoop-administrator-interview-questions/..

Unknown said...


Thanks for providing this informative information. it is very useful you may also refer-
http://www.s4techno.com/blog/2016/08/12/interview-questions-of-hive/

Anonymous said...

root@dell:/usr/local/hadoop-2.6.5/sbin# start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
[Fatal Error] :-1:-1: Premature end of file.
17/10/30 20:37:48 FATAL conf.Configuration: error parsing conf mapred-site.xml
org.xml.sax.SAXParseException; Premature end of file.
.
.
.
.
.
.
.
.
.
.
Starting namenodes on []
localhost: starting namenode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-namenode-dell.out
localhost: [Fatal Error] :-1:-1: Premature end of file.
localhost: starting datanode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-datanode-dell.out
localhost: [Fatal Error] :-1:-1: Premature end of file.
[Fatal Error] :-1:-1: Premature end of file.
17/10/30 20:38:00 FATAL conf.Configuration: error parsing conf mapred-site.xml
org.xml.sax.SAXParseException; Premature end of file.
.
.
.
.
.
.
.

starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop-2.6.5/logs/yarn-root-resourcemanager-dell.out
[Fatal Error] :-1:-1: Premature end of file.
localhost: starting nodemanager, logging to /usr/local/hadoop-2.6.5/logs/yarn-root-nodemanager-dell.out
localhost: [Fatal Error] :-1:-1: Premature end of file.
root@dell:/usr/local/hadoop-2.6.5/sbin# jps
25999 Jps
root@dell:/usr/local/hadoop-2.6.5/sbin#

In this case what should be done?