文章目录
系统要求
系统：Centos7
Hadoop版本：2.8.0
安装JAVA环境
Hadoop许多组件都是JAVA语言编写的，安装JAVA环境才能正常运行，所以JAVA环境必不可少。
JAVA版本：1.8.0_65
百度网盘下载地址：
链接：https://pan.baidu.com/s/1gtwyeF9UVnMfnb5x3bQNKw
提取码：7mk5
可使用WINSCP 文件上传至Centos 7Linux
使用命令：
tar -zxvf jdk-8u65-linux-x64.t.gz
[root@zhang jvm]# tar -zxvf jdk-8u65-linux-x64.t.gz
jdk1.8.0_65
进入刚上传的路径，输入./java -version查询版本，显示如下，表示安装成功
cd /usr/lib/jvm/jdk1.8.0_65/bin
[root@zhang bin]# ./java -versionjava version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM)64-Bit Server VM (build 25.65-b01, mixed mode)
配置环境变量PATH（重要）
使用VIM打开，编辑profile
vi /etc/profile
在末端添加以下代码，如果跟我路径一致，可直接复制，不一样的话，要注意自己安装路径，即上面上传解压的路径。就是这一句代码修改为自己的路径：export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_65，后面2个直接复制即可。
exportJAVA_HOME=/usr/lib/jvm/jdk1.8.0_65exportCLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
exportPATH=$JAVA_HOME/bin:$PATH
使用wq!保存退出
修改完成要使环境变量生效，要source下
[root@zhang bin]# source /etc/profile
[root@zhang bin]#
测试下环境变量是否起作用，能显示版本号就成功
[root@zhang bin]# java -version
java version "1.8.0_65"Java(TM)SE Runtime Environment(build 1.8.0_65-b17)
Java HotSpot(TM)64-Bit Server VM(build 25.65-b01, mixed mode)[root@zhang bin]#
安装Hadoop的HDFS和Yarn
Hadoop版本号：hadoop-2.8.0
网盘下载地址：
链接：https://pan.baidu.com/s/1QJ8x8NJGkV4j6WD-JiNJ6Q
提取码：xopm
使用WINSCP上传至Centos 7 Linux系统
选择上传目标路径，我这里选择:/opt/tools
[root@zhang bin]# cd /opt/tools/[root@zhang tools]# ls
apache-flume-1.9.0-bin.tar.gz hadoop-2.8.0.tar.gz
解压
tar -zxvf hadoop-2.8.0.tar.gz
使用CP 命令复制解压后的文件到 /opt/server/,如下复制后并更名为hadoop-2.8.0
[root@zhang server]# ls
flume hadoop-2.8.0
Hadoop安装路径如下：后面设置环境变量需要，要记住这个地址
[root@zhang server]# cd hadoop-2.8.0/[root@zhang hadoop-2.8.0]# ls
bin etc lib LICENSE.txt NOTICE.txt sbin
data include libexec logs README.txt share
[root@zhang hadoop-2.8.0]# pwd
/opt/server/hadoop-2.8.0
配置hadoop-env.sh，末尾增添JAVA_HOME 的路径
root@zhang hadoop]# vi /opt/server/hadoop-2.8.0/etc/hadoop/hadoop-env.sh
exportJAVA_HOME=/usr/lib/jvm/jdk1.8.0_65
配置hadoop-env.sh，末尾增添JAVA_HOME 的路径
配置 core-site.xml
在配置文件夹/opt/server/hadoop-2.8.0/etc/hadoop/
使用VIM编辑
vi core-site.xml
<configuration><property><name>fs.defaultFS</name><value>hdfs://zhang:9000/</value></property><property><name>hadoop.tmp.dir</name><value>/opt/server/hadoop-2.8.0/data/</value></property></configuration>
注意：
第一个property中的Value:hdfs://zhang:9000/，修改主机名，比如我的主机名为zhang,端口号9000是固定的，特别注意下 主机名，可以使用
hostname 查询下，错了容易出现问题
第二个property中的Value:hadoop的安装路径，opt/server/hadoop-2.8.0/data/，数据存储在data文件夹里
配置 hdfs-site.xml
<configuration><property><name>dfs.replication</name><value>1</value></property><property></configuration>
配置 mapred-site.xml
打开mapred-site.xml,编辑以下内容，直接复制
<configuration><property><name>mapreduce.framework.name</name><value>yarn</value></property></configuration>
配置 yarn-site.xml
打开yarn-site.xml,编辑以下内容，直接复制
<configuration><property><name>yarn.resourcemanager.hostname</name><value>zhang</value></property><property><name>yarn.nodemanager.aux-servies</name><value>mapreduce_shuffle</value></property></configuration>
第一个property的value为主机名，要更改为自己的主机名，另一个直接复制，无需更改
配置Hadoop环境变量，跟JAVA环境变量一致，只需在profile文件末尾添加即可
vi /etc/profile
exportHADOOP_HOME=/opt/server/hadoop-2.8.0exportPATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
exportHDFS_NAMENODE_USER=root
exportHDFS_DATANODE_USER=root
exportHDFS_SECONDARYNAMENODE_USER=root
exportYARN_RESOURCEMANAGER_USER=root
exportYARN_NODEMANAGER_USER=root
用户名可以按实际写，Hadoop_Home 为安装路径，家目录，根据实际路径填写即可。
使用SSH免密登录
设置SSH，可以免密操作，无需频繁输入密码，尤其是多个节点操作时需要免密登录
ssh-keygen -t rsa
一路ENTER确认即可
[root@zhang hadoop]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key(/root/.ssh/id_rsa):/root/.ssh/id_rsa already exists.Overwrite(y/n)? y
Enter passphrase(empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in/root/.ssh/id_rsa.
Your public key has been saved in/root/.ssh/id_rsa.pub.
The key fingerprint is:SHA256:Hf31D+MXPXuDrXUG8AKWri+FxKVWTozOPchnmn/s6NA root@zhang
The key's randomart image is:+---[RSA2048]----+| o ||.=o ||=O= o .||X+*o +.o|| oS*o..*oo||+o. oo+=|| ooE...=*||.o..o oo+|| o+o..|+----[SHA256]-----+[root@zhang hadoop]#
将公钥追加到authorized_keys文件中去
[root@zhang hadoop]# cd /root/.ssh/[root@zhang .ssh]# ls
authorized_keys id_rsa id_rsa.pub known_hosts
[root@zhang .ssh]#
[root@zhang .ssh]# cat id_rsa.pub >> authorized_keys
[root@zhang .ssh]# cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAgPnSvMuak7tITg4osi1paBHZracOfD5OVvgWnIocmv2p+ShYv4GCjGXEEJRJPRzAf+6J25lTEaQQbZf6pFiS72lYbZltPQ8E2Mb00wFE/cuyJLoI49dgAISTF+1PYZqcuFIbPPM+voZYPp+8HxJqXkBZj7XHvxPDNrU2k13lNrRsgvLEXRHVgRIjJaZIar9yx6Tm21XyY88Eqnw0WhoIFw/TqTMEu7dYcBKNfgoDTRcg/xTyDqAGpZOxiYHtGNqp7F3R2ngw5QbzBo0O3MrsifX9jmSeueDMWk92xIDR1cXPYwrArXASwFjQBszl8aMx9XZ8a+Rs1Xjt2DQ0VRzb [email protected]
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAisWpCpt9fK6NcMKEKz2eD1yP6BEOOrDjWcQhWXjmuHjhrbVsjZEUf6xr/ub3k/21Q0QeoFfK64fg7nF1aukS9jxNB1L8uBt5f95YnUPiocz5cLp7XzkMf4qInP5aXCzLkHFvvi7NThYsrfp3yUGMKt/8VgJSXVJTCPssG2+sMBkevZSDIFm5o1BKUmMuQWu82T7g4GXOw3F0y+ZrHM+oniF1fliaSNj7r1wUM8YgKxeHYrLn9JdkH/eil7Ay777MZGfO1zLDwsJjQVhiEY6cnsFKDy4nAutI/ep6nr+o0dCPLrUkSAfSPa2JFFb3F9YJN3GJ0jNZjgfqQRinNRGX [email protected]
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDN88B3ux29tH0qRQbaXM5bpEEmqRusQ2gbNEaSxLGeONJeG78cUnkmFeLdgKznZI99sDwhlhwL4jnrL8jWvOou+II6KcRJ6k45myDpg9GNqQvc836r7v4hMetIAeHjtRAOZjIT7CuT+o+9BviF7xrUerC9vyXn+8lUfNEZW+TnmTaV26bVxeET/RX13z8em5wJzWRYRtSXCVBB3cthpwBm7oxzoXdOj/gtGWpq5bg3iDgeEPWSwor1//Jh+hlYTCvvfTpCpX9QLuB+WoWTX5nsUgqd5FDME2tsPFQCoAstr17iFgUyr4fhuVU7/emC+cbYzqK15Iey4o1UK0ghNWAH root@zhang[root@zhang .ssh]#
使用SSH 主机名 登录，无需密码
[root@zhang .ssh]# ssh zhang
Last login: Tue Jul 1612:40:562024[root@zhang ~]#
启动HADOOP
初始化
这一步很重要
如果出现 INFO common.Storage: Storage directory /opt/server/hadoop-2.8.0/data/dfs/name has been successfully formatted. 为成功格式化
[root@zhang ~]# hadoop namenode -format
DEPRECATED: Use ofthis script to execute hdfs command is deprecated.
Instead use the hdfs command for it.24/07/1613:00:59INFO namenode.NameNode:STARTUP_MSG:/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: user = root
STARTUP_MSG: host = zhang/192.168.66.128
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.8.0
STARTUP_MSG: classpath = /opt/server/hadoop-2.8.0/etc/hadoop:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/activation-1.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/asm-3.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/avro-1.7.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-collections-3.2.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-io-2.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-net-3.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/curator-client-2.7.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/curator-framework-2.7.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/gson-2.2.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/guava-11.0.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/hadoop-annotations-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/hadoop-auth-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/htrace-core4-4.0.1-incubating.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/httpclient-4.5.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/httpcore-4.4.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jcip-annotations-1.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jettison-1.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jsch-0.1.51.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/json-smart-1.1.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jsr305-3.0.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/junit-4.11.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/paranamer-2.3.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/xz-1.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/zookeeper-3.4.6.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/hadoop-common-2.8.0-tests.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/hadoop-common-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/hadoop-nfs-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/htrace-core4-4.0.1-incubating.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/okhttp-2.4.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/okio-1.4.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/hadoop-hdfs-2.8.0-tests.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/hadoop-hdfs-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/hadoop-hdfs-client-2.8.0-tests.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/hadoop-hdfs-client-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/hadoop-hdfs-native-client-2.8.0-tests.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/hadoop-hdfs-native-client-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/activation-1.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/asm-3.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/commons-math-2.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/curator-client-2.7.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/curator-test-2.7.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/fst-2.24.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/guava-11.0.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/guice-3.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/objenesis-2.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/xz-1.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-api-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-client-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-common-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-registry-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-server-common-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/asm-3.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/guice-3.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/javax.inject-1.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/junit-4.11.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/xz-1.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.0-tests.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.0.jar:/opt/server/hadoop-2.8.0/contrib/capacity-scheduler/*.jar:/opt/server/hadoop-2.8.0/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r 91f2b7a13d1e97be65db92ddabc627cc29ac0009; compiled by 'jdu' on 2017-03-17T04:12Z
STARTUP_MSG: java = 1.8.0_65
************************************************************/24/07/1613:00:59INFO namenode.NameNode: registered UNIX signal handlers for[TERM,HUP,INT]24/07/1613:00:59INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid:CID-8b3bc4bb-baa8-4fb4-b71a-9a26a6626ef5
24/07/1613:01:01INFO namenode.FSEditLog: Edit logging is async:false24/07/1613:01:01INFO namenode.FSNamesystem: KeyProvider:null24/07/1613:01:01INFO namenode.FSNamesystem: fsLock is fair:true24/07/1613:01:01INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled:false24/07/1613:01:02INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=100024/07/1613:01:02INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true24/07/1613:01:02INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.00024/07/1613:01:02INFO blockmanagement.BlockManager: The block deletion will start around 2024 Jul 1613:01:0224/07/1613:01:02INFO util.GSet: Computing capacity for map BlocksMap
24/07/1613:01:02INFO util.GSet:VM type =64-bit
24/07/1613:01:02INFO util.GSet:2.0% max memory 966.7MB=19.3MB24/07/1613:01:02INFO util.GSet: capacity =2^21=2097152 entries
24/07/1613:01:02INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false24/07/1613:01:02INFO blockmanagement.BlockManager: defaultReplication =124/07/1613:01:02INFO blockmanagement.BlockManager: maxReplication =51224/07/1613:01:02INFO blockmanagement.BlockManager: minReplication =124/07/1613:01:02INFO blockmanagement.BlockManager: maxReplicationStreams =224/07/1613:01:02INFO blockmanagement.BlockManager: replicationRecheckInterval =300024/07/1613:01:02INFO blockmanagement.BlockManager: encryptDataTransfer =false24/07/1613:01:02INFO blockmanagement.BlockManager: maxNumBlocksToLog =100024/07/1613:01:02INFO namenode.FSNamesystem: fsOwner =root(auth:SIMPLE)24/07/1613:01:02INFO namenode.FSNamesystem: supergroup = supergroup
24/07/1613:01:02INFO namenode.FSNamesystem: isPermissionEnabled =true24/07/1613:01:02INFO namenode.FSNamesystem:HAEnabled:false24/07/1613:01:02INFO namenode.FSNamesystem: Append Enabled:true24/07/1613:01:02INFO util.GSet: Computing capacity for map INodeMap
24/07/1613:01:02INFO util.GSet:VM type =64-bit
24/07/1613:01:02INFO util.GSet:1.0% max memory 966.7MB=9.7MB24/07/1613:01:02INFO util.GSet: capacity =2^20=1048576 entries
24/07/1613:01:02INFO namenode.FSDirectory: ACLs enabled?false24/07/1613:01:02INFO namenode.FSDirectory: XAttrs enabled?true24/07/1613:01:02INFO namenode.NameNode: Caching file names occurring more than 10 times
24/07/1613:01:02INFO util.GSet: Computing capacity for map cachedBlocks
24/07/1613:01:02INFO util.GSet:VM type =64-bit
24/07/1613:01:02INFO util.GSet:0.25% max memory 966.7MB=2.4MB24/07/1613:01:02INFO util.GSet: capacity =2^18=262144 entries
24/07/1613:01:02INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct =0.999000012874603324/07/1613:01:02INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes =024/07/1613:01:02INFO namenode.FSNamesystem: dfs.namenode.safemode.extension =3000024/07/1613:01:02INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets =1024/07/1613:01:02INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users =1024/07/1613:01:02INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes =1,5,2524/07/1613:01:03INFO namenode.FSNamesystem: Retry cache on namenode is enabled
24/07/1613:01:03INFO namenode.FSNamesystem: Retry cache will use 0.03of total heap and retry cache entry expiry time is 600000 millis
24/07/1613:01:03INFO util.GSet: Computing capacity for map NameNodeRetryCache
24/07/1613:01:03INFO util.GSet:VM type =64-bit
24/07/1613:01:03INFO util.GSet:0.029999999329447746% max memory 966.7MB=297.0KB24/07/1613:01:03INFO util.GSet: capacity =2^15=32768 entries
Re-format filesystem in Storage Directory /opt/server/hadoop-2.8.0/data/dfs/name ?(Y or N) y
24/07/1613:01:06INFO namenode.FSImage: Allocated newBlockPoolId:BP-108476542-192.168.66.128-172110606684924/07/1613:01:06INFO common.Storage: Storage directory /opt/server/hadoop-2.8.0/data/dfs/name has been successfully formatted.24/07/1613:01:06INFO namenode.FSImageFormatProtobuf: Saving image file /opt/server/hadoop-2.8.0/data/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
24/07/1613:01:07INFO namenode.FSImageFormatProtobuf: Image file /opt/server/hadoop-2.8.0/data/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 321 bytes saved in0 seconds.24/07/1613:01:07INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >=024/07/1613:01:07INFO util.ExitUtil: Exiting with status 024/07/1613:01:07INFO namenode.NameNode:SHUTDOWN_MSG:/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at zhang/192.168.66.128
************************************************************/
启动进程
start-dfs.sh
start-yarn.sh
[root@zhang hadoop]# start-dfs.sh
Starting namenodes on [zhang]zhang: starting namenode, logging to /opt/server/hadoop-2.8.0/logs/hadoop-root-namenode-zhang.out
localhost: starting datanode, logging to /opt/server/hadoop-2.8.0/logs/hadoop-root-datanode-zhang.out
Starting secondary namenodes [0.0.0.0]0.0.0.0: starting secondarynamenode, logging to /opt/server/hadoop-2.8.0/logs/hadoop-root-secondarynamenode-zhang.out
[root@zhang hadoop]#
[root@zhang hadoop]# start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/server/hadoop-2.8.0/logs/yarn-root-resourcemanager-zhang.out
localhost: starting nodemanager, logging to /opt/server/hadoop-2.8.0/logs/yarn-root-nodemanager-zhang.out
[root@zhang hadoop]#
关闭Hadoop
stop-all.sh
也可以
stop-yarn.sh
start-dfs.sh
如果出现namenode 或datanode无法启动，删除/opt/server/hadoop-2.8.0/l所有version文件
验证安装结果
使用jps命令，显示如下6个进程即安装成功
[root@zhang hadoop]# jps
60034 NodeManager
59907 ResourceManager
59654 SecondaryNameNode
59466 DataNode
60618 Jps
59307 NameNode
[root@zhang hadoop]#
最后可以应用flume采集日志数据，实时地传输到这个Hadoop文件系统，再次验证这个系统的安装是否准确无误，可以参考我其他文章，如有问题，欢迎一起学习讨论！
Flume+Hadoop大数据采集部署
参考文章
在Linux虚拟机下安装Hadoop的保姆级教程
本文转载自: https://blog.csdn.net/weixin_42732268/article/details/140453031
版权归原作者 终有一刻 所有， 如有侵权，请联系我们删除。
版权归原作者 终有一刻 所有， 如有侵权，请联系我们删除。