ubuntu16.04 安装单机Hadoop&HIVE

2020-07-12 原文

硬件准备

Win10上安装virtualBox,采用桥接模式，网卡为wireless

虚拟机地址为，192.168.1.188
```
cat /etc/issue
       Ubuntu 16.04.2 LTS \n \l
```

其他已安装（除去jdk外非必需）

安装了redis/usr/bin/redis-server,redis-cli
      安装了java user/bin/java
      安装了sonar/usr/local/sonar/sonarqube-5.6.6/bin/linux-x86-    
             64/sonar.shstart 默认监听9000端口
      安装了MysqL server && client 用户名密码root
      安装了PHP 7.0
      安装了apache2  apachectl -v 2.4.10
      安装了szrz小工具
      安装了jenkins service jenkins start，默认监听8080端口 用户名密码
                        tongbo

软件准备

jdk

root@ubuntu:/usr/bin#  /usr/local/java/jdk1.8.0_121/bin/java -version
    java version "1.8.0_121"
    Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
    Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13,mixed mode)
    root@ubuntu:/usr/bin#
    root@ubuntu:/home/tb# /usr/local/java/jdk1.8.0_121/bin/jps
    2050 Jps
    1533 jenkins.war

hadoop官网下载地址

可以利用szrz小工具，我是解压后hadoop安装目录为：

/home/tb/tbdown/hadoop-2.8.2

tar zxf  hadoop-2.8.2.tar.gz

root@ubuntu:/home/tb/tbdown# ls
dump.rdb      hadoop-2.8.2-src         hadoop-2.8.2.tar.gz  Nginx-1.8.1.tar.gz
hadoop-2.8.2  hadoop-2.8.2-src.tar.gz  Nginx-1.8.1          spider111

验证hadoop是否安装成功

root@ubuntu:/home/tb/tbdown/hadoop-2.8.2# ./bin/hadoop version
Hadoop 2.8.2
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 66c47f2a01ad9637879e95f80c41f798373828fb
Compiled by jdu on 2017-10-19T20:39Z
Compiled with protoc 2.5.0
From source with checksum dce55e5afe30c210816b39b631a53b1d
This command was run using /home/tb/tbdown/hadoop-2.8.2/share/hadoop/common/hadoop-common-2.8.2.jar

修改配置

注意所有操作都在hadoop安装目录（/home/tb/tbdown/hadoop-2.8.2/）进行

如有需要改动ip配置，重启网卡 /etc/init.d/networking restart

vim /etc/hosts
添加一行
127.0.0.1 tb001

然后需要修改配置文件，在hadoop安装目录的./etc/hadoop下

root@ubuntu:/home/tb/tbdown/hadoop-2.8.2/etc/hadoop# ls
capacity-scheduler.xml      hadoop-policy.xml        kms-log4j.properties        ssl-client.xml.example
configuration.xsl           hdfs-site.xml            kms-site.xml                ssl-server.xml.example
container-executor.cfg      httpfs-env.sh            log4j.properties            yarn-env.cmd
core-site.xml               httpfs-log4j.properties  mapred-env.cmd              yarn-env.sh
hadoop-env.cmd              httpfs-signature.secret  mapred-env.sh               yarn-site.xml
hadoop-env.sh               httpfs-site.xml          mapred-queues.xml.template
hadoop-metrics2.properties  kms-acls.xml             mapred-site.xml.template
hadoop-metrics.properties   kms-env.sh               slaves

修改配置文件前，最好先备份一下原始的配置文件

vim hadoop-env.sh

需要更改第一行的 export java home ，如果已经配置了全局java，则无需更改，否则使JAVA_HOME =/your java.jdk...*/

将simple配置文件复制一份自己定义的，yarn是用来hadoop的资源管理系统，

root@ubuntu:/home/tb/tbdown/hadoop-2.8.2/etc/hadoop# cp mapred-site.xml.template mapred-site.xml



vim mapred-site.xml

<configuration>
 <property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
 </property>
</configuration>


vim core-site.xml

如果没有设置hosts里面的对应，以下的tb001可以设置为localhost，fs.defaultFS是用来设置hadoop的文件系统，默认就是hdfs了。这样客户端可以通过8020端口来连接namenode服务，hdfs的守护进程也会通过该属性确定主机和端口

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs:://tb001:8020</value>
    </property>
</configuration>

vim hdfs-site.xml

第一个参数配置为副本数，由于是单机，先为1，默认应该是3，但是我们是伪分布式，设置为3会在block做副本的时候报错，就是说无法将某块复制到3个datanode上。
另外关于副本数量，可以通过hodoop fs -ls命令看到副本的数量
第二三个参数配置为两个目录配置，配置好后启动hadoop会自动创建，默认为/tmp/,如果是虚拟机，请一定设置非/tmp路径

<property>
  <name>dfs.replication</name>
  <value>1</value>
 </property>
 <property>
  <name>dfs.namenode.name.dir</name>
  <value>/home/tb/hadoop/dfs/name</value>
 </property>
 <property>
  <name>dfs.datanode.data.dir</name>
  <value>/home/tb/hadoop/dfs/data</value>


vim yarn-site.xml
  <property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_value</value>
 </property>



vim slaves

如果是单机，loalhost或者修改对应hosts的值都可以
默认为localhost

启动服务

启动namenode ，启动之前先进行格式化，第一次进行时运行，如果后期再运行，将格式化所有数据

root@ubuntu:/home/tb/tbdown/hadoop-2.8.2# ./bin/hadoop namenode -format

格式化后对应目录将会有以下变化（根据你设置的hdfs-site.xml第一个设置dfs路径的）：多了dfs目录及以下
（启动datanode会创建第二个配置文件中的目录，见下文）

root@ubuntu:/home/tb/hadoop# pwd
/home/tb/hadoop
root@ubuntu:/home/tb/hadoop# tree ./
./
└── dfs
    └── name
        └── current
            ├── fsimage_0000000000000000000
            ├── fsimage_0000000000000000000.md5
            ├── seen_txid
            └── VERSION

3 directories,4 files

启动namenode

root@ubuntu:/home/tb/tbdown/hadoop-2.8.2# ./sbin/hadoop-daemon.sh start namenode
starting namenode,logging to /home/tb/tbdown/hadoop-2.8.2/logs/hadoop-root-namenode-ubuntu.out
root@ubuntu:/home/tb/tbdown/hadoop-2.8.2#

如何知道namenode有没有成功呢？看下面没有对应进程，说明没有成功，下面我们调试错误

/usr/local/java/jdk1.8.0_121/bin/jps
1533 jenkins.war
3215 Jps

调试错误

没有成功，可以查看log


root@ubuntu:/home/tb/tbdown/hadoop-2.8.2# cd logs/
root@ubuntu:/home/tb/tbdown/hadoop-2.8.2/logs# ls
hadoop-root-namenode-ubuntu.log  hadoop-root-namenode-ubuntu.out  SecurityAuth-root.audit
root@ubuntu:/home/tb/tbdown/hadoop-2.8.2/logs# tail -f hadoop-root-namenode-ubuntu.log
  at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:682)
  at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:905)
  at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:884)
  at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1610)
  at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1678)
2017-11-04 16:42:22,937 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2017-11-04 16:42:22,939 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ubuntu/127.0.1.1
************************************************************/

找到问题所在：

java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): hdfs:://tb001:8020 has no authority.

看问题是没有授权，解决办法如下[可以参考文末官方文档]，ssh主要是方便主节点直接登录操作其他子节点，无需单独登录到子节点逐一管理

apt-get install ssh
sudo apt-get install pdsh
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

再次执行启动namenode就可以了，

root@ubuntu:/home/tb/tbdown/hadoop-2.8.2# ./sbin/hadoop-daemon.sh start namenode
starting namenode,logging to /home/tb/tbdown/hadoop-2.8.2/logs/hadoop-root-namenode-ubuntu.out
root@ubuntu:/home/tb/tbdown/hadoop-2.8.2# /usr/local/java/jdk1.8.0_121/bin/jps
6658 NameNode
6731 Jps
1550 jenkins.war

namenode已经启动成功，继续启动其他吧

当然也可以把dfs的namenode和datanode 一起执行，执行 sbin/start-dfs.sh即可

root@ubuntu:/home/tb/tbdown/hadoop-2.8.2# sbin/start-dfs.sh
Starting namenodes on [tb001]
tb001: namenode running as process 6658. Stop it first.
localhost: starting datanode,logging to /home/tb/tbdown/hadoop-2.8.2/logs/hadoop-root-datanode-ubuntu.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode,logging to /home/tb/tbdown/hadoop-2.8.2/logs/hadoop-root-secondarynamenode-ubuntu.out
root@ubuntu:/home/tb/tbdown/hadoop-2.8.2# /usr/local/java/jdk1.8.0_121/bin/jps
6658 NameNode
7235 Jps
7124 SecondaryNameNode
6942 Datanode
1550 jenkins.war
root@ubuntu:/home/tb/tbdown/hadoop-2.8.2#

停止


root@ubuntu:/home/tb/tbdown/hadoop-2.8.2# sbin/stop-dfs.sh
Stopping namenodes on [tb001]
tb001: stopping namenode
localhost: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
root@ubuntu:/home/tb/tbdown/hadoop-2.8.2#

启动yarn

root@ubuntu:/home/tb/tbdown/hadoop-2.8.2# sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager,logging to /home/tb/tbdown/hadoop-2.8.2/logs/yarn-root-resourcemanager-ubuntu.out
localhost: starting nodemanager,logging to /home/tb/tbdown/hadoop-2.8.2/logs/yarn-root-nodemanager-ubuntu.out
root@ubuntu:/home/tb/tbdown/hadoop-2.8.2# /usr/local/java/jdk1.8.0_121/bin/jps
9252 SecondaryNameNode
8918 NameNode
10039 Jps
9066 Datanode
1550 jenkins.war
9631 ResourceManager
root@ubuntu:/home/tb/tbdown/hadoop-2.8.2#

验证成功

yarn ：http://192.168.1.188:8088/cluster

hdfs ：http://192.168.1.188:50070/dfshealth.html#tab-overview

自动安装部署

ambari
minos（小米开源hadoop部署）
choudear manager（收费）

一键打包兼容

hdp
cdh4 OR cdh5