图片 6

检测当前系统下的jdk安装情况:

环境:系统CentOS6.6;Hadoop版本:1.0.3;java运行环境:jdk1.6

[root@Centos 桌面]# rpm -qa | grep java

单节点配置过程:

tzdata-java-2012j-1.el6.noarch

1.配置系统ssh:hadoop在运行过程中会用访问ssh服务,将ssh服务设置成无密码访问,这样hadoop在访问ssh服务的时候就不需要人工手动输入密码了:

java-1.7.0-openjdk-1.7.0.9-2.3.4.1.el6_3.x86_64

detail:
step 1:生成密钥
[hjchaw@localhost ~]$ ssh-keygen -t rsa -P “”
[hjchaw@localhost ~]$ cat ~/.ssh/id_rsa.pub >>
~/.ssh/authorized_keys
step 2:测试ssh ,如果ssh成功连接,说明配置ssh配置成功
[hjchaw@localhost ~]$ ssh localhost
如果ssh访问还提示输入密码:一般是.ssh路径访问权限问题,把权限设置成700,配置的时候注意。

java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64

2.hadoop配置过程:
step1:hadoop-env.xml配置,修改其中的JAVA_HOME:如:JAVA_HOME=/usr/local/jdk
step2:core-site.xml文件配置:
<configuration>
  NO1:配置hadoop数据存放路径
  <property>
    <name>hadoop.tmp.dir</name>
   
<value>/home/hjchaw/hadoop-datastore/hadoop-${user.name}</value>
    <description>The name of the default file system.  Either
the
      literal string “local” or a host:port for NDFS.
    </description>
    <final>true</final>
  </property>

 

 NO2:设置fs名称
 <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
    <description>The name of the default file system.  Either
the
      literal string “local” or a host:port for NDFS.
    </description>
    <final>true</final>
  </property>
</configuration>

卸载当前所有的jdk:

step3:配置hdfs-site.xml
 
  <!– file system properties –>
  <property>
    <name>dfs.name.dir</name>
    <value>${hadoop.tmp.dir}/dfs/name</value>
    <description>Determines where on the local filesystem the DFS
name node
      should store the name table.  If this is a comma-delimited list
      of directories then the name table is replicated in all of the
      directories, for redundancy. </description>
    <final>true</final>
  </property>

[root@Centos 桌面]# rpm -e –nodeps tzdata-java-2012j-1.el6.noarch

  <property>
    <name>dfs.data.dir</name>
    <value>${hadoop.tmp.dir}/dfs/data</value>
    <description>Determines where on the local filesystem an DFS
data node
       should store its blocks.  If this is a comma-delimited
       list of directories, then data will be stored in all named
       directories, typically on different devices.
       Directories that do not exist are ignored.
    </description>
    <final>true</final>
  </property>

[root@Centos 桌面]# rpm -e –nodeps
java-1.7.0-openjdk-1.7.0.9-2.3.4.1.el6_3.x86_64

  <property>
    <name>dfs.replication</name>
    <value>1</value>
    <final>true</final>
  </property>
step4: mapred-site.xml 配置
<configuration>
     <property>
         <name>mapred.job.tracker</name>
         <value>localhost:9001</value>
     </property>
</configuration>
以上几步是hadoop单节点,伪分布式配置。
3.hadoop启动:
可以将hadoop/bin设置到PATH路径中
setup1:格式化文件系统:
   [hjchaw@localhost bin]$  hadoop namenode -format
  
12/05/27 04:25:19 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = localhost.localdomain/127.0.0.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.0.3
STARTUP_MSG:   build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
1335192; compiled by ‘hortonfo’ on Tue May  8 20:31:25 UTC 2012
************************************************************/
12/05/27 04:25:19 INFO util.GSet: VM type       = 32-bit
12/05/27 04:25:19 INFO util.GSet: 2% max memory = 19.33375 MB
12/05/27 04:25:19 INFO util.GSet: capacity      = 2^22 = 4194304
entries
12/05/27 04:25:19 INFO util.GSet: recommended=4194304, actual=4194304
12/05/27 04:25:20 INFO namenode.FSNamesystem: fsOwner=hjchaw
12/05/27 04:25:20 INFO namenode.FSNamesystem: supergroup=supergroup
12/05/27 04:25:20 INFO namenode.FSNamesystem: isPermissionEnabled=true
12/05/27 04:25:20 INFO namenode.FSNamesystem:
dfs.block.invalidate.limit=100
12/05/27 04:25:20 INFO namenode.FSNamesystem: isAccessTokenEnabled=false
accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
12/05/27 04:25:20 INFO namenode.NameNode: Caching file names occuring
more than 10 times
12/05/27 04:25:21 INFO common.Storage: Image file of size 112 saved in 0
seconds.
12/05/27 04:25:21 INFO common.Storage: Storage directory
/home/hjchaw/hadoop-datastore/hadoop-hjchaw/dfs/name has been
successfully formatted.
12/05/27 04:25:21 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at
localhost.localdomain/127.0.0.1
************************************************************/
step2:启动hadoop:
[hjchaw@localhost bin]$ start-all.sh
starting namenode, logging to
/opt/hadoop/hadoop-1.0.3/libexec/../logs/hadoop-hjchaw-namenode-localhost.localdomain.out
localhost: starting datanode, logging to
/opt/hadoop/hadoop-1.0.3/libexec/../logs/hadoop-hjchaw-datanode-localhost.localdomain.out
localhost: starting secondarynamenode, logging to
/opt/hadoop/hadoop-1.0.3/libexec/../logs/hadoop-hjchaw-secondarynamenode-localhost.localdomain.out
starting jobtracker, logging to
/opt/hadoop/hadoop-1.0.3/libexec/../logs/hadoop-hjchaw-jobtracker-localhost.localdomain.out
localhost: starting tasktracker, logging to
/opt/hadoop/hadoop-1.0.3/libexec/../logs/hadoop-hjchaw-tasktracker-localhost.localdomain.out

[root@Centos 桌面]# rpm -e –nodeps
java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64

如果看到以上结果信息,那么configuration is OK,now!

 

4.尝试使用hadoop命令行接口操作文件系统: 如:新建一个文件夹:
[hjchaw@localhost bin]$ hadoop fs -mkdir input
查看文件:
[hjchaw@localhost bin]$ hadoop fs -ls
Found 1 items
drwxr-xr-x   – hjchaw supergroup          0 2012-05-27 04:26
/user/hjchaw/input

复查是否卸载完全:无输出则卸载干净:

到此hadoop单节点,伪分布式配置结束。

[root@Centos 桌面]# rpm -qa | grep java

图片 1

[root@Centos 桌面]#

 

 

 

安装自己下载的jdk并配置环境变量,详尽参考“linux学习/NO.7”

 

 

······················································· JDK安装完成······························································

 

······················································· hadoop的安装开始··················································

 

1.在hadoop的conf目录下配置
hadoop-env.sh   core-site.xml   hdfs-site.xml   mapred-site.xml ,

首先解压hadoop.

            1.1 在hadoop-env.sh里的配置hadoop的JDK环境

            ———————————————

            [root@Centos ~]# cd hadoop-1.2.1/

            [root@Centos hadoop-1.2.1]# cd
conf

            [root@Centos conf]# vi hadoop-env.sh

            ———————————————

            配置信息如下:

                export JAVA_HOME=/root/jdk1.8.0_65   

 

            1.2 在core-site.xml里的配置hadoop的HDFS地址及端口号

            ————————————————

            [root@Centos conf]# vi core-site.xml

            ————————————————

            配置信息如下:

                <configuration>

                    <property>

                        <name>fs.default.name</name>

                        <value>hdfs://localhost:9000</value>

                    </property>

                </configuration>

 

            1.3 在hdfs-site.xml里的配置hadoop的HDFS的配置

            ————————————————-

            [root@Centos conf]# vi hdfs-site.xml

            ————————————————-

            配置信息如下:

                <configuration>

                    <property>

                    <name>dfs.replication</name>

                    <value>1</value>

                    </property>

                </configuration>

 

            1.4 在mapred-site.xml里的配置hadoop的HDFS的配置

            ————————————————-

            [root@Centos conf]# vi mapred-site.xml

            ——————————————–

            配置信息如下:

                <configuration>

                    <property>         

                      <name>mapred.job.tracker</name>

                      <value>localhost:9001</value>

                    </property>

                </configuration>

图片 2

 

2.ssh免密码登录


[root@Centos conf]# cd /root

[root@Centos ~]# ssh-keygen -t
rsa

输出:

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

ed:48:64:29:62:37:c1:e9:3d:84:bf:ad:4e:50:5e:66 root@Centos

The key’s randomart image is:

+–[ RSA 2048]—-+

|     ..o         |

|      +…       |

|    o.++= E      |

|   . o.B+=       |

|      . S+.      |

|       o.o.      |

|        o..      |

|       ..        |

|       ..        |

+—————–+

c[root@Centos ~]# cd .ssh

[root@Centos .ssh]# ls

id_rsa  id_rsa.pub

[root@Centos .ssh]# cp id_rsa.pub  authorized_keys

[root@Centos .ssh]# ls

authorized_keys  id_rsa  id_rsa.pub

[root@Centos .ssh]# ssh localhost

The authenticity of host ‘localhost (::1)’ can’t be established.

RSA key fingerprint is 3f:84:db:2f:53:a9:09:a6:61:a2:3a:82:80:6c:af:1a.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added ‘localhost’ (RSA) to the list of known hosts.


    验证免密码登录


[root@Centos ~]# ssh localhost

Last login: Sun Apr  3 23:19:51 2016 from localhost

 

[root@Centos ~]# ssh localhost

Last login: Sun Apr  3 23:20:12 2016 from localhost

 

Connection to localhost closed.

[root@Centos ~]#

—————————-SSH免密码登录设置成功—————————-

 

 

格式化HDFS:

[root@Centos ~]# cd  /root/hadoop-1.2.1/

[root@Centos hadoop-1.2.1]#  bin/hadoop namenode -format

输出:

16/04/03 23:24:12 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = java.net.UnknownHostException: Centos: Centos:
unknown error

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 1.2.1

STARTUP_MSG:   build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r
1503152; compiled by ‘mattf’ on Mon Jul 22 15:23:09 PDT 2013

STARTUP_MSG:   java = 1.8.0_65

************************************************************/

16/04/03 23:24:13 INFO util.GSet: Computing capacity for map BlocksMap

16/04/03 23:24:13 INFO util.GSet: VM type       = 64-bit

16/04/03 23:24:13 INFO util.GSet: 2.0% max memory = 1013645312

16/04/03 23:24:13 INFO util.GSet: capacity      = 2^21 = 2097152 entries

16/04/03 23:24:13 INFO util.GSet: recommended=2097152, actual=2097152

16/04/03 23:24:15 INFO namenode.FSNamesystem: fsOwner=root

16/04/03 23:24:15 INFO namenode.FSNamesystem: supergroup=supergroup

16/04/03 23:24:15 INFO namenode.FSNamesystem: isPermissionEnabled=true

16/04/03 23:24:15 INFO namenode.FSNamesystem:
dfs.block.invalidate.limit=100

16/04/03 23:24:15 INFO namenode.FSNamesystem: isAccessTokenEnabled=false
accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)

16/04/03 23:24:15 INFO namenode.FSEditLog:
dfs.namenode.edits.toleration.length = 0

16/04/03 23:24:15 INFO namenode.NameNode: Caching file names occuring
more than 10 times

16/04/03 23:24:17 INFO common.Storage: Image file
/tmp/hadoop-root/dfs/name/current/fsimage of size 110 bytes saved in 0
seconds.

16/04/03 23:24:18 INFO namenode.FSEditLog: closing edit log: position=4,
editlog=/tmp/hadoop-root/dfs/name/current/edits

16/04/03 23:24:18 INFO namenode.FSEditLog: close success: truncate to 4,
editlog=/tmp/hadoop-root/dfs/name/current/edits

16/04/03 23:24:18 INFO common.Storage: Storage directory
/tmp/hadoop-root/dfs/name has been successfully formatted.

16/04/03 23:24:18 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at java.net.UnknownHostException:
Centos: Centos: unknown error

************************************************************/


此时格式化节点报错:Centos: unknown
error——–紧接着下一步配置


[root@Centos hadoop-1.2.1]# vi /etc/hosts

    配置信息如下:

    127.0.0.1   localhost Centos


再一次进行格式化       


[root@Centos hadoop-1.2.1]# vi /etc/hosts

[root@Centos hadoop-1.2.1]# bin/hadoop namenode -format

具体情况如下图:

图片 3

 

—————————namenode格式化成功——————————

启动hadoop

        关闭防火墙命令行  # service
iptables stop

      启动hadoop集群命令行  #
start-all.sh

      关闭hadoop集群命令行  #
stop-all.sh

关闭防火墙:# service iptables stop

图片 4

启动hadoop集群 :

[root@Centos hadoop-1.2.1]# bin/start-all.sh

输出:

starting namenode, logging to
/root/hadoop-1.2.1/libexec/../logs/hadoop-root-namenode-Centos.out

localhost: starting datanode, logging to
/root/hadoop-1.2.1/libexec/../logs/hadoop-root-datanode-Centos.out

localhost: starting secondarynamenode, logging to
/root/hadoop-1.2.1/libexec/../logs/hadoop-root-secondarynamenode-Centos.out

starting jobtracker, logging to
/root/hadoop-1.2.1/libexec/../logs/hadoop-root-jobtracker-Centos.out

localhost: starting tasktracker, logging to
/root/hadoop-1.2.1/libexec/../logs/hadoop-root-tasktracker-Centos.out

 

验证集群是否正常启动—-5个节点在列表中则启动成功

再次验证启动项目

[root@Centos hadoop-1.2.1]# cd  /root/mahout-distribution-0.6/

[root@Centos mahout-distribution-0.6]# jps

30692 SecondaryNameNode

30437 NameNode

31382 Jps

30903 TaskTracker

30775 JobTracker

30553 DataNode

[root@Centos mahout-distribution-0.6]# jps

30692 SecondaryNameNode

31477 Jps

30437 NameNode

30903 TaskTracker

30775 JobTracker

30553 DataNode

[root@Centos mahout-distribution-0.6]# cd 
/home/hadoop-1.2.1

关闭hadoop集群

[root@Centos hadoop-1.2.1]# bin/stop-all.sh

stopping jobtracker

localhost: stopping tasktracker

stopping namenode

localhost: stopping datanode

localhost: stopping secondarynamenode

[root@Centos hadoop-1.2.1]#

图片 5

 

————————hadoop伪分布式安装成功————————

mahout的安装

            1.解压安装mahout :解压到hadoop目录中

                [root@Centos hadoop-1.2.1]#  tar zxvf
mahout-distribution-0.6.tar.gz

            2.配置环境变量

                    export HADOOP_HOME=/root/hadoop-1.2.1

                    export HADOOP_CONF_DIR=/root/hadoop-1.2.1/conf

                    export
MAHOUT_HOME=/root/hadoop-1.2.1/mahoutdistribution-0.6

                    export
MAHOUT_CONF_DIR=/root/hadoop-1.2.1/mahoutdistribution-0.6/conf

                    export
PATH=$PATH:$MAHOUT_HOME/conf:$MAHOUT_HOME/bin

            3.测试mahout的启动

配置环境变量过程如下图:

 图片 6

————————————–mahout安装成————————————

 

 

 

admin

相关文章

发表评论

电子邮件地址不会被公开。 必填项已用*标注