0
0

Hadoop集群进行map词频统计

一、首先新建虚拟机

二、配置静态IP

  1.     1、首先查看虚拟网络编辑器 查看起始IP
  3.             ![](https://img-blog.csdnimg.cn/9f9b0fc72fb345e7bb58ae06de2909c9.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_17,color_FFFFFF,t_70,g_se,x_16)
  5.      2.1、修改静态IP
  7.             输入指令：vi /etc/sysconfig/network-scripts/ifcfg-ens33
  9.             修改BOOTPROTO=static
  11.             增加IPADDR、NETWASK、GATEWAY、DNS1
  13.             ![](https://img-blog.csdnimg.cn/de82dc4bfe9b4d98acfd63b1687b5462.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)
  15.             ![](https://img-blog.csdnimg.cn/dedfb9effb6449febeb1be7310e4dd9a.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)
  17.      2.2、输入指令：vi /etc/sysconfig/network增加以下两条
  19.            ![](https://img-blog.csdnimg.cn/53733028997b461da21e3c07bc9752d3.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16) 
  21.              ![](https://img-blog.csdnimg.cn/988e426cef9a4441997763cd903abad4.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_9,color_FFFFFF,t_70,g_se,x_16)
  23.      2.3、输入指令：vi /etc/hosts 添加上IP和主机名
  25.             ![](https://img-blog.csdnimg.cn/42b69a5c48784cb29b4e03f239560e69.png)
  27.      2.4、输入：reboot 重启虚拟机

三、安装JDK

  1.     3.1、在opt目录下创建module、jdk文件夹
  3.             输入命令：cd /opt/
  5.             输入命令：mkdir module
  7.             输入命令：mkdir jdk
  9.             输入命令：mkdir hadoop
  11.     3.2、卸载当前jdk
  13.             输入命令：java -version 查看当前jdk版本
  15.             输入命令：yum remove java* 卸载所有jdk
  17.             ![](https://img-blog.csdnimg.cn/ab75cbda67b24ff0a239fb2dd17602d7.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)
  19.      3.3、使用FileZilla链接虚拟机
  21.              将jdk压缩包上传到hadoop102的opt/jdk目录下、
  23.              hadoop压缩包上传到hadoop102的opt/hadoop目录下。
  25.             ![](https://img-blog.csdnimg.cn/59cce46b63e04b818203b7469f62ddb5.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)
  27.      3.4、解压压缩包到制定目录
  29.             输入指令：cd /opt/jdk
  31.             输入指令：tar -zxvf jdk（jdk压缩包） -C /opt/module/
  33.             ![](https://img-blog.csdnimg.cn/2b19a2f71f3d49f9842ce931280591bb.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)
  35.      3.5、配置profile文件并让其生效
  37.             输入指令：pwd 查看当前目录
  39.             输入指令：vi /etc/profile  在文件末尾添加JAVA_HOME
  41.             ![](https://img-blog.csdnimg.cn/09db62fec3734338a08b2ea1449f028e.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_9,color_FFFFFF,t_70,g_se,x_16)
  43.              输入指令：source /etc/profile
  45.              输入指令：java -version
  47.             ![](https://img-blog.csdnimg.cn/3d0d1f7b1b9e4d79b00ed641182948e4.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)

四、安装hadoop

  1.     4.1、解压hadoop到指定目录
  3.             切换到根目录
  5.             输入指令：mkdir kkb
  7.             输入指令：cd kkb
  9.             输入指令：mkdir install
  11.             输入指令：cd /opt/hadoop
  13.             输入指令：tar -zxvf hadoop（hadoop压缩包） -C /kkb/install
  15.            ![](https://img-blog.csdnimg.cn/e5e4602bec1848e5b25b7207c7fb3531.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)
  17.      4.2、配置profile文件并使其生效
  19.             输入指令：vi /etc/profile 配置HADOOP_HOME环境
  21.             ![](https://img-blog.csdnimg.cn/bcdc5e2ddcb84f52903b8588af60908c.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)
  23.             输入指令：source /etc/profile
  25.             输入指令：hadoop version
  27.             ![](https://img-blog.csdnimg.cn/0eb3abb3e11e4674b51036b445719f15.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)

五、克隆出hadoop103、hadoop104，并向hadoop102一样步骤修改静态IP

六、配置ssh免密登录

  1.      6.1、以102为例配置ssh
  3.             输入指令：cd ~/.ssh
  5.             输入指令：ssh-keygen -t rsa
  7.             连续输入三个回车，生成密匙       
  9.     6.2、分发密匙，优先分发给自己，再分发给103、104
  11.             输入指令：ssh-copy-id 192.168.88.130
  13.             输入指令：ssh-copy-id 192.168.88.131
  15.             输入指令：ssh-copy-id 192.168.88.132
  17.      6.3、在103、104上按照6.1-6.2的步骤配置ssh
  19.             ![](https://img-blog.csdnimg.cn/f5aa11e2a5504575a17c5be21c25b1db.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_9,color_FFFFFF,t_70,g_se,x_16)
  21.              ![](https://img-blog.csdnimg.cn/8470272efc7b472fb2cbc9e1d1db0c3f.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_10,color_FFFFFF,t_70,g_se,x_16)

七、配置集群分发脚本xsync

  1.     7.1、在/usr/local/bin目录下创建xsync文件
  3.             输入指令：vi /usr/local/bin/xsync
  5.     7.2、xsync内容文件如下：
  1. #!/bin/bash
  2. #1 获取输入参数个数，如果没有参数，直接退出
  3. pcount=$#
  4. if((pcount==0)); then
  5. echo no args;
  6. exit;
  7. fi
  9. #2 获取文件名称
  10. p1=$1
  11. fname=`basename $p1`
  12. echo fname=$fname
  14. #3 获取上级目录到绝对路径
  15. pdir=`cd -P $(dirname $p1); pwd`
  16. echo pdir=$pdir
  18. #4 获取当前用户名称
  19. user=`whoami`
  21. #5 循环
  22. for((host=103; host<105; host++)); do
  23.         #echo $pdir/$fname $user@hadoop$host:$pdir
  24.         echo --------------- hadoop$host ----------------
  25.         rsync -rvl $pdir/$fname $user@hadoop$host:$pdir
  26. done
  1.     7.3、修改文件权限
  3.             输入指令：chomd a+x xsync

八、hadoop3的集群配置

  1.     8.1、执行checknative
  3.             输入指令：cd /kkb/install/hadoop-3.1.4/
  5.             输入指令：bin/hadoop checknative
  7.             ![](https://img-blog.csdnimg.cn/12f58f690a964b8aa43a165501d13d2c.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)
  9.     8.2、安装openssl-deve1
  11.             输入指令：yum -y install openssl-deve1
  13.     8.3、修改dfs、yarn配置文件
  15.             输入指令：cd /kkb/install/hadoop-3.1.4/etc/hadoop
  17.             输入指令：vim /hadoop-env.sh   在末尾添加以下内容
  1. export JAVA_HOME=/kkb/install/jdk1.8.0_162
  1.             输入指令：vim core-site.xml  在标签内添加以下内容
  1. <configuration>
  2.     <property>
  3.         <name>fs.defaultFS</name>
  4.         <value>hdfs://hadoop102:8020</value>
  5.     </property>
  6.     <property>
  7.         <name>hadoop.tmp.dir</name>
  8.         <value>/kkb/install/hadoop-3.1.4/hadoopDatas/tempDatas</value>
  9.     </property>
  10.     <!--  缓冲区大小，实际工作中根据服务器性能动态调整；默认值4096 -->
  11.     <property>
  12.         <name>io.file.buffer.size</name>
  13.         <value>4096</value>
  14.     </property>
  15.     <!--  开启hdfs的垃圾桶机制，删除掉的数据可以从垃圾桶中回收，单位分钟；默认值0 -->
  16.     <property>
  17.         <name>fs.trash.interval</name>
  18.         <value>10080</value>
  19.     </property>
  20. </configuration>
  1.              输入指令：vim /hdfs-site.xml
  1. <configuration>
  2. <!-- NameNode存储元数据信息的路径，实际工作中，一般先确定磁盘的挂载目录，然后多个目录用，进行分割   -->
  3.     <!--   集群动态上下线
  4.     <property>
  5.         <name>dfs.hosts</name>
  6.         <value>/kkb/install/hadoop-3.1.4/etc/hadoop/accept_host</value>
  7.     </property>
  8.     <property>
  9.         <name>dfs.hosts.exclude</name>
  10.         <value>/kkb/install/hadoop-3.1.4/etc/hadoop/deny_host</value>
  11.     </property>
  12.      -->
  13.      <property>
  14.             <name>dfs.namenode.secondary.http-address</name>
  15.             <value>hadoop102:9868</value>
  16.     </property>
  17.     <property>
  18.         <name>dfs.namenode.http-address</name>
  19.         <value>hadoop102:9870</value>
  20.     </property>
  21.     <!-- namenode保存fsimage的路径 -->
  22.     <property>
  23.         <name>dfs.namenode.name.dir</name>
  24.         <value>file:///kkb/install/hadoop-3.1.4/hadoopDatas/namenodeDatas</value>
  25.     </property>
  26.     <!--  定义dataNode数据存储的节点位置，实际工作中，一般先确定磁盘的挂载目录，然后多个目录用，进行分割  -->
  27.     <property>
  28.         <name>dfs.datanode.data.dir</name>
  29.         <value>file:///kkb/install/hadoop-3.1.4/hadoopDatas/datanodeDatas</value>
  30.     </property>
  31.     <!-- namenode保存editslog的目录 -->
  32.     <property>
  33.         <name>dfs.namenode.edits.dir</name>
  34.         <value>file:///kkb/install/hadoop-3.1.4/hadoopDatas/dfs/nn/edits</value>
  35.     </property>
  36.     <!-- secondarynamenode保存待合并的fsimage -->
  37.     <property>
  38.         <name>dfs.namenode.checkpoint.dir</name>
  39.         <value>file:///kkb/install/hadoop-3.1.4/hadoopDatas/dfs/snn/name</value>
  40.     </property>
  41.     <!-- secondarynamenode保存待合并的editslog -->
  42.     <property>
  43.         <name>dfs.namenode.checkpoint.edits.dir</name>
  44.         <value>file:///kkb/install/hadoop-3.1.4/hadoopDatas/dfs/nn/snn/edits</value>
  45.     </property>
  46.     <property>
  47.         <name>dfs.replication</name>
  48.         <value>3</value>
  49.     </property>
  50.     <property>
  51.         <name>dfs.permissions.enabled</name>
  52.         <value>false</value>
  53.     </property>
  54.         <property>
  55.         <name>dfs.blocksize</name>
  56.         <value>134217728</value>
  57.     </property>
  58. </configuration>
  1.             输入指令：vim mapred-site.xml
  1. <configuration>
  2. <property>
  3.         <name>mapreduce.framework.name</name>
  4.         <value>yarn</value>
  5.     </property>
  6.     <property>
  7.         <name>mapreduce.job.ubertask.enable</name>
  8.         <value>true</value>
  9.     </property>
  10.     <property>
  11.         <name>mapreduce.jobhistory.address</name>
  12.         <value>hadoop102:10020</value>
  13.     </property>
  14.     <property>
  15.         <name>mapreduce.jobhistory.webapp.address</name>
  16.         <value>hadoop102:19888</value>
  17.     </property>
  18.         <property>
  19.         <name>yarn.app.mapreduce.am.env</name>
  20.         <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
  21.     </property>
  22.     <property>
  23.         <name>mapreduce.map.env</name>
  24.         <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
  25.     </property>
  26.     <property>
  27.         <name>mapreduce.reduce.env</name>
  28.         <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
  29.     </property>
  30. </configuration>
  1.             输入指令：vi yarn-site.xml
  1. <configuration>
  3. <!-- Site specific YARN configuration properties -->
  4. <property>
  5.        <name>yarn.resourcemanager.hostname</name>
  6.         <value>hadoop102</value>
  7.     </property>
  8.     <property>
  9.         <name>yarn.nodemanager.aux-services</name>
  10.         <value>mapreduce_shuffle</value>
  11.     </property>
  12.     <!-- 如果vmem、pmem资源不够，会报错，此处将资源监察置为false -->
  13.     <property>
  14.         <name>yarn.nodemanager.vmem-check-enabled</name>
  15.         <value>false</value>
  16.     </property>
  17.     <property>
  18.         <name>yarn.nodemanager.pmem-check-enabled</name>
  19.         <value>false</value>
  20.     </property>
  21. </configuration>
  1.             输入指令：vi workers
  1. hadoop102
  2. hadoop103
  3. hadoop104
  1.     8.4、创建文件存放目录
  3.             输入指令：mkdir -p /kkb/install/hadoop-3.1.4/hadoopDatas/tempDatas
  5.             输入指令：mkdir -p /kkb/install/hadoop-3.1.4/hadoopDatas/namenodeDatas
  7.             输入指令：mkdir -p /kkb/install/hadoop-3.1.4/hadoopDatas/datanodeDatas
  9.             输入指令：mkdir -p /kkb/install/hadoop-3.1.4/hadoopDatas/dfs/nn/edits
  11.             输入指令：mkdir -p /kkb/install/hadoop-3.1.4/hadoopDatas/dfs/snn/name
  13.             输入指令：mkdir -p /kkb/install/hadoop-3.1.4/hadoopDatas/dfs/nn/snn/edits
  15.     8.5、使用xsync分发配置文件               
  17.             输入指令：xsync hadoop-3.1.4  102在install目录下将hadoop分发给103、104

九、启动hdfs、yarn

  1.     9.1、在102上格式化集群（只能格式一次、不能频繁格式）
  3.             输入指令：hdfs namenode -format
  5.     9.2、在102的hadoop-3.1.4目录下启动dfs、yarn
  7.             输入指令：sbin/start-dfs.sh
  9.             输入指令：sbin/start-yarn.sh
  11.     9.3、jps命令查看启动进程
  13.             ![](https://img-blog.csdnimg.cn/cb57e951621b4506afb0834f85e9206f.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_6,color_FFFFFF,t_70,g_se,x_16)![](https://img-blog.csdnimg.cn/ab14fe00cc614a4ca92ffe2512d036ff.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_7,color_FFFFFF,t_70,g_se,x_16)
  15.              ![](https://img-blog.csdnimg.cn/c5001bdb7a434964b2732ffd745599c5.png)
  17.      9.4、验证集群是否启动成功
  19.             在浏览器打开：192.168.88.130:8088
  21.             在浏览器打开：192.168.88.130:9870
  23.              ![](https://img-blog.csdnimg.cn/ebe2bcbdcc3a4c95a49531d32d360afc.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)
  25.              ![](https://img-blog.csdnimg.cn/8b6cd92534b34e21a3cab1cfcee9d0ed.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)

十、在Windows中配置hadoop

  1.     10.1、修改windows的hosts文件
  3.               地址：C:\Windows\System32\drivers\etc\hosts
  5.           ![](https://img-blog.csdnimg.cn/52917d929c504a4f8df2305bf25fcfa1.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)   
  7.   10.2、配置Windows本中配置hadoop环境
  9.              将集群所用的hadoop-3.1.4.tar.gz解压到一个没有中文、空格的目录下
  11.   10.3、配置hadoop的环境变量
  13.             ![](https://img-blog.csdnimg.cn/a18668b854b44e1a80d92bc7eded68b3.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)
  15.              ![](https://img-blog.csdnimg.cn/6d7c6135f15b4c5d88b3549d52448ab1.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_9,color_FFFFFF,t_70,g_se,x_16)
  17.    10.4、将下图的hadoop.dll文件拷贝到C:\\Windows\System32
  19.                 ![](https://img-blog.csdnimg.cn/819b5bc96a7b47a1abfb5a28b5be807e.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)         
  21.     10.5、将hadoop集群的一下5个配置文件core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml、workers，拷贝到windows下hadoop的C:\hadoop-3.1.4\etc\hadoop目录下
  23.     10.6、打开cmd运行hadoop命令
  25.            ![](https://img-blog.csdnimg.cn/cb615145d04349d0aefc810de28ed55c.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_11,color_FFFFFF,t_70,g_se,x_16)

十一、安装maven 

  1.     11.1、下载安装包 apache-maven-3.6.1-bin.zip  并解压到某目录、配置环境变量
  3.             ![](https://img-blog.csdnimg.cn/2273d796aaa749b0805d77439eed8b58.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)
  5.             ![](https://img-blog.csdnimg.cn/a48efdeb546d4aef9849f42ce1e16c69.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_10,color_FFFFFF,t_70,g_se,x_16)
  7.     11.2、cmd中运行mvn -v
  9.            ![](https://img-blog.csdnimg.cn/436eba03149440e08a666d3aea3fa962.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)         
  11.     11.3、找到maven解压的目录，找到settings.xml文件，添加以下内容
  13.             ![](https://img-blog.csdnimg.cn/5bf6ee3a169c4f42b0d4926f90e18e61.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)
  15.              ![](https://img-blog.csdnimg.cn/6c392c7058aa48009ce4546997f06f2f.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)
  17.       11.4、打开IDEA，新建一个maven工程，配置pom文件，内容如下：
  1. <properties>
  2.         <hadoop.version>3.1.4</hadoop.version>
  3.     </properties>
  5.     <dependencies>
  6.         <dependency>
  7.             <groupId>org.apache.hadoop</groupId>
  8.             <artifactId>hadoop-client</artifactId>
  9.             <version>${hadoop.version}</version>
  10.         </dependency>
  11.         <dependency>
  12.             <groupId>org.apache.hadoop</groupId>
  13.             <artifactId>hadoop-common</artifactId>
  14.             <version>${hadoop.version}</version>
  15.         </dependency>
  17.         <dependency>
  18.             <groupId>org.apache.hadoop</groupId>
  19.             <artifactId>hadoop-hdfs</artifactId>
  20.             <version>${hadoop.version}</version>
  21.         </dependency>
  23.         <dependency>
  24.             <groupId>org.apache.hadoop</groupId>
  25.             <artifactId>hadoop-mapreduce-client-core</artifactId>
  26.             <version>${hadoop.version}</version>
  27.         </dependency>
  28.         <!-- https://mvnrepository.com/artifact/junit/junit -->
  29.         <dependency>
  30.             <groupId>junit</groupId>
  31.             <artifactId>junit</artifactId>
  32.             <version>4.11</version>
  33.             <scope>test</scope>
  34.         </dependency>
  35.         <dependency>
  36.             <groupId>org.testng</groupId>
  37.             <artifactId>testng</artifactId>
  38.             <version>RELEASE</version>
  39.         </dependency>
  40.         <dependency>
  41.             <groupId>log4j</groupId>
  42.             <artifactId>log4j</artifactId>
  43.             <version>1.2.17</version>
  44.         </dependency>
  45.     </dependencies>
  46.     <build>
  47.         <plugins>
  48.             <plugin>
  49.                 <groupId>org.apache.maven.plugins</groupId>
  50.                 <artifactId>maven-compiler-plugin</artifactId>
  51.                 <version>3.0</version>
  52.                 <configuration>
  53.                     <source>1.8</source>
  54.                     <target>1.8</target>
  55.                     <encoding>UTF-8</encoding>
  56.                     <!--   <verbal>true</verbal>-->
  57.                 </configuration>
  58.             </plugin>
  59.             <plugin>
  60.                 <groupId>org.apache.maven.plugins</groupId>
  61.                 <artifactId>maven-shade-plugin</artifactId>
  62.                 <version>2.4.3</version>
  63.                 <executions>
  64.                     <execution>
  65.                         <phase>package</phase>
  66.                         <goals>
  67.                             <goal>shade</goal>
  68.                         </goals>
  69.                         <configuration>
  70.                             <minimizeJar>true</minimizeJar>
  71.                         </configuration>
  72.                     </execution>
  73.                 </executions>
  74.             </plugin>
  75.         </plugins>
  76.     </build>

十二、词频统计程序实现

  1.     12.1、编写mapper类
  1. package wordcount;
  3. import org.apache.hadoop.io.IntWritable;
  4. import org.apache.hadoop.io.LongWritable;
  5. import org.apache.hadoop.io.Text;
  6. import org.apache.hadoop.mapreduce.Mapper;
  8. import java.io.IOException;
  9. import java.util.ArrayList;
  10. import java.util.List;
  12. public class MyMapper extends Mapper <LongWritable, Text,Text, IntWritable>{
  13.     @Override
  14.     protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
  15.         //获得当前行的数据
  16.         String line = value.toString();
  18.         //获得一个个的单词
  19.             String lineBuffer = line;
  21.             String[] keys = new String[]{" ", "\t", "    ", ".", "（", "）", "(", ")"};
  23.             for (String k : keys){
  24.             lineBuffer = lineBuffer.replace(k, ",");
  25.         }
  27.         String[] wordsBuffer = lineBuffer.split(",");
  29.         List<String> words = new ArrayList<>();
  31.         for (String w : wordsBuffer){
  32.             if (!w.equals("")){
  33.                 words.add(w);
  34.             }
  35.         }
  37.         //每个单词编程kv对
  38.         for (String word : words) {
  39.             //将kv对输出出去
  40.             context.write(new Text(word),new IntWritable(1));
  41.         }
  43.     }
  44. }
  1.     12.2、编写Reducer类
  1. package wordcount;
  3. import org.apache.hadoop.io.IntWritable;
  4. import org.apache.hadoop.io.Text;
  5. import org.apache.hadoop.mapreduce.Reducer;
  7. import java.io.IOException;
  9. public class MyReducer extends Reducer<Text, IntWritable,Text,IntWritable> {
  10.     //bear,List(2,3,3)
  11.     @Override
  12.     protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
  13.         int sum=0;
  14.         for (IntWritable value : values) {
  15.             int count = value.get();
  16.             sum +=count;
  17.         }
  19.         context.write(key,new IntWritable(sum));
  20.     }
  21. }
  1.     12.3、组装main程序
  1. package wordcount;
  3. import org.apache.hadoop.conf.Configuration;
  4. import org.apache.hadoop.conf.Configured;
  5. import org.apache.hadoop.fs.Path;
  6. import org.apache.hadoop.io.IntWritable;
  7. import org.apache.hadoop.io.Text;
  8. import org.apache.hadoop.mapreduce.Job;
  9. import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
  10. import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
  11. import org.apache.hadoop.util.Tool;
  12. import org.apache.hadoop.util.ToolRunner;
  14. public class WordCount extends Configured implements Tool {
  15. public static void main(String[] args) throws Exception {
  16. int run = ToolRunner.run(new Configuration(), new WordCount(), args); // 集群代码
  17. System.exit(run);
  19. }
  20. @Override
  21. public int run(String[] args) throws Exception {
  22. Job job = Job.getInstance(super.getConf(), "wordcount");
  23.         job.setJarByClass(WordCount.class);
  25.         job.setInputFormatClass(TextInputFormat.class);
  27.         TextInputFormat.addInputPath(job, new Path(args[0]));
  29.         job.setMapperClass(MyMapper.class);
  30.         job.setMapOutputKeyClass(Text.class);
  31.         job.setMapOutputValueClass(IntWritable.class);
  33.         job.setReducerClass(MyReducer.class);
  34.         job.setOutputKeyClass(Text.class);
  35.         job.setOutputValueClass(IntWritable.class);
  37.         job.setOutputFormatClass(TextOutputFormat.class);
  38.         TextOutputFormat.setOutputPath(job, new Path(args[1]));
  40.         job.setNumReduceTasks(Integer.parseInt(args[2]));
  42.         boolean b = job.waitForCompletion(true);
  43.         return b ? 0 : 1;
  45.     }
  46. }
  1.     12.4、将程序打包、点击maven的package
  3.            ![](https://img-blog.csdnimg.cn/2dcbc6cb2d3d4798bc9232932430b158.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)

十三、在集群中实现

  1.     13.1、使用FileZilla链接hadoop102
  3.                找到打包文件，和测试文件，一起上传到hadoop102
  5.              ![](https://img-blog.csdnimg.cn/2ca791df8d40411fb619e28eabc95528.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)       
  7.     13.2、将测试文件上传到hdfs
  9.                在hadoop-3.1.4目录下
  11.                输入指令：bin/hdfs dfs -mkdir -p /test-wrh  在hdfs上创建test-wrh文件夹
  13.                输入指令：bin/hdfs dfs -put 测试文件地址  /test-wrh/  将测试文件上传到test-wrh
  15.              ![](https://img-blog.csdnimg.cn/94ba732d10704ecdb085471fbd19a8bd.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)  
  17.              ![](https://img-blog.csdnimg.cn/48277b092321420ba133fc8136fa387d.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)
  19.      13.3、在IDEA中拷贝地址
  21.                  ![](https://img-blog.csdnimg.cn/7a0c004ee11e4a858255f0224554afd2.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)   
  23.      13.4、运行程序
  25.                在包含程序的目录中：
  27.                输入指令：hadoop jar jar包名  Reference /输入路径  /输出路径  3个节点

13.5、将结果从hdfs上下载

  1.             ![](https://img-blog.csdnimg.cn/4e716e2a62d341eea3859f10c4a1c676.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_12,color_FFFFFF,t_70,g_se,x_16)
  3.             输入指令：hadoop fs -get /输出路径/part-r-00000 /下载路径

  1.       13.6、查看结果
  3.                  输入指令：vim part-r-00000
  5.             ![](https://img-blog.csdnimg.cn/6bc02f11f7664a858a9ad913210bc552.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBAd2VpeGluXzQ3MjMxNzEz,size_10,color_FFFFFF,t_70,g_se,x_16)
  9.              
  15.                 
  19.              
  21.                         
  23.                           
  27.
标签： hadoop 大数据 big data
本文转载自: https://blog.csdn.net/weixin_47231713/article/details/122156621
版权归原作者 陈信宇是大聪明 所有， 如有侵权，请联系我们删除。
登录后发布评论

“Hadoop集群进行map词频统计”的评论:

还没有评论
关于作者
...
overfit同步小助手
文章同步
相关阅读

网络安全法-网络运行安全

使用selenium/drissionpage时如何阻止chrome自动跳转http到https

docker desktop 里部署的Open WebUI 管理员密码忘记了的处理方法

在ubuntu20.04中搭建onsite比赛运行环境

利用开源的低代码表单设计器FcDesigner高效管理和渲染复杂表单结构

Kafka学习笔记

【前端】浏览器输入url到页面呈现发生了什么？

文章导航
没有找到文章内容分段。。。