0
0

大数据技术之Hadoop-3.3.4高可用安装部署操作文档

一、集群部署规划

环境Linux+centos8

注意：NameNode和SecondaryNameNode不要安装在同一台服务器

注意：ResourceManager也很消耗内存，不要和NameNode、SecondaryNameNode配置

因为环境不允许，所以有点重叠的部署组件
主机名namenodedatanodeJobHistoryServerResourcemanagerNodeManagerZKJournalNodeZKFCmaster√√√√√salve1√√√√√salve2√√√√√salve3√√salve4√√

二、zookeeper安装部署以及脚本使用说明

2.0 将zookeeper安装包到/opt/software

[appview@master ~]$ cd /opt/software/
[appview@master software]$
[appview@master software]$ tar -zxvf apache-zookeeper-3.7.1-bin.tar.gz -C /opt/moudle/

2.1配置服务器编号

[appview@master software]$ cd /opt/moudle/zookeeper-3.7.1/
#### 创建zData目录
[appview@master software]$ mkdir zData
[appview@master zData]$ vim myid
###### 添加 1 如下
[appview@master zData]$ cat myid
1
[appview@master zData]$
​

2.2配置zoo.cfg文件

[appview@master conf]$ pwd
/opt/moudle/zookeeper-3.7.1/conf
[appview@master conf]$ ll
总用量 12
-rw-r--r-- 1 appview appview  535 5月   7 2022 configuration.xsl
-rw-r--r-- 1 appview appview 3435 5月   7 2022 log4j.properties
-rw-rw-r-- 1 appview appview 1246 3月  29 08:45 zoo_sample.cfg
[appview@master conf]$
[appview@master conf]$ mv zoo_sample.cfg zoo.cfg
[appview@master conf]$ vim zoo.cfg

########## 修改数据路径
dataDir=/opt/moudle/zookeeper-3.7.1/zData          
​
########## 添加以下内容
#######################cluster##########################
server.1=master:2888:3888
server.2=salve1:2888:3888
server.3=salve2:2888:3888

2.3分发安装包

[appview@master conf]$ xsync-1-3  /opt/moudle/zookeeper-3.7.1

修改slave1、salve2路径/opt/moudle/zookeeper-3.7.1/zData/myid,分别改为对应的2、3

2.4添加环境变量

[appview@master software]$ vim ~/.bashrc

添加以下内容

#ZK_HOME export ZK_HOME=/opt/moudle/zookeeper-3.7.1 export PATH=$PATH:$ZK_HOME/bin

2.5 一键启动zookeeper脚本

[appview@master bin]$ vim  zk.sh
脚本内容如下：
#!/bin/bash
case $1 in
"start"){
      for i in  master salve1 salve2
          do
          echo --------zookeeper-3.7.1 $i -----------------
                ssh $i "/opt/moudle/zookeeper-3.7.1/bin/zkServer.sh start"
          done
};;
"stop"){
      for i in master salve1 salve2
          do
          echo --------zookeeper-3.7.1 $i -----------------
                ssh $i "/opt/moudle/zookeeper-3.7.1/bin/zkServer.sh stop"
          done
};;
"status"){
      for i in master salve1 salve2
          do
          echo --------zookeeper-3.7.1 $i -----------------
                ssh $i "/opt/moudle/zookeeper-3.7.1/bin/zkServer.sh status"
          done
};;
esac
启动zookeeper服务
#####启动
[appview@master bin]$ zk.sh start   
#####停止
[appview@master bin]$ zk.sh stop
#####状态
[appview@master bin]$ zk.sh status

2.6 脚本说明以及使用

rsync是一个强大的文件同步工具，可以在远程或本地之间同步文件和目录，并且在同步时可以实现增量传输，只传输变更的部分，从而节约带宽和时间
### 基本格式
rsync [选项] 源目录/文件
-a  归档模式（保留目录机构及权限）
-v  显示详细输出信息
-r 递归复制整个目录
-u 仅在源文件比目标文件新或者不存在时进行复制
-z 传输时压缩文件
-e 指定远程shell程序
--delete 删除目标目录中不存在于源文件中的文件
--exclude = PATTERN:排除匹配指定模式的文件或目录
在家目录下创建bin目标，在里面放脚本可以在任意地方调用
分发脚本xsync如下
[appview@master ~]$ cd bin/
[appview@master bin]$ vim xsync
#### 内容如下，根据自己需要分发的几个节点进行修改
​
#!/bin/bash
# 判断参数个数
if [ $# -lt 1 ]
then
   echo Not Enough Arguement!
   exit;
fi
# 遍历集群所有机器
for host in master salve1 salve2 salve3 salve4
do
  echo ================ $host ==========================
​
  # 遍历所有目录，挨个发送
  for file in $@
  do
  # 判断文件是否存在
  if [ -e $file ]
  then
  # 获取父目录
  pdir=$(cd -P $(dirname $file); pwd)
  # 获取当前文件的名称
  fname=$(basename $file)
  ssh $host "mkdir -p $pdir"
  rsync -av $pdir/$fname  $host:$pdir
  else
    echo $file does not exists!!!
  fi
  done
done
xcall脚本如下
#!/bin/bash
​
for host in master salve1 salve2 salve3 salve4
do
    echo ----------- $host -----------------
    ssh $host "$*"
done
​
xcall-1-3脚本如下
[appview@master bin]$ vim xcall-1-3
########内容如下：
#!/bin/bash
​
for host in master salve1 salve2
do
    echo ----------- $host -----------------
    ssh $host "$*"
done

三、Hadoop安装部署

3.1将hadoop-3.3.4.tar.gz安装包到/opt/software/
[appview@master ~]$ cd /opt/software/
[appview@master software]$
[appview@master software]$ tar -zxvf hadoop-3.3.4.tar.gz -C /opt/moudle/
3.2将Hadoop添加到环境变量
[appview@master software]$ vim ~/.bashrc
### 添加以下内容
#HADOOP_HOME
export HADOOP_HOME=/opt/moudle/hadoop-3.3.4
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

保存后，将环境变量分发以下，或者每台执行以上操作也可，

### 环境变量分发
[appview@master software]$ xsync ~/.bashrc
### 执行以下命令使环境变量生效
[appview@master software]$ source ~/.bashrc

四、配置集群

1.核心配置文件

1.1配置core-site.xml
[appview@master software]$ cd $HADOOP_HOME/etc/hadoop
[appview@master hadoop]$ vim core-site.xml
文件内容如下：
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at
​
    http://www.apache.org/licenses/LICENSE-2.0
​
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
​
<!-- Put site-specific property overrides in this file. -->
​
<configuration>
<!-- 把多个 NameNode 的地址组装成一个集群 hdpcluster -->
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://hdpcluster</value>
        </property>
<!-- 指定 hadoop 运行时产生文件的存储目录 -->
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/opt/moudle/hadoop-3.3.4/data/tmp</value>
        </property>
 <!-- 指定 zkfc 要连接的 zkServer 地址 -->
        <property>
                <name>ha.zookeeper.quorum</name>
                <value>master:2181,salve1:2181,salve2:2181</value>
        </property>
<!-- NN 连接 JN 重试次数，默认是 10 次 -->
    <property>
       <name>ipc.client.connect.max.retries</name>
       <value>20</value>
    </property>
<!-- 重试时间间隔，默认 1s -->
   <property>
      <name>ipc.client.connect.retry.interval</name>
      <value>30000</value>
   </property>
<!-- 配置HDFS网页登录使用的静态用户为appview -->
    <property>
        <name>hadoop.http.staticuser.user</name>
        <value>appview</value>
</property>
<!-- 配置该appview(superUser)允许通过代理访问的主机节点 -->
    <property>
        <name>hadoop.proxyuser.appview.hosts</name>
        <value>*</value>
</property>
<!-- 配置该appview(superUser)允许通过代理用户所属组 -->
    <property>
        <name>hadoop.proxyuser.appview.groups</name>
        <value>*</value>
</property>
<!-- 配置该appview(superUser)允许通过代理的用户-->
    <property>
        <name>hadoop.proxyuser.appview.users</name>
        <value>*</value>
</property>
</configuration>
2.HDFS配置文件
2.1配置hdfs-site.xml
[appview@master hadoop]$ vim hdfs-site.xml
文件内容如下：
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at
​
    http://www.apache.org/licenses/LICENSE-2.0
​
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
​
<!-- Put site-specific property overrides in this file. -->
​
<configuration>
        <!-- NameNode 数据存储目录 -->
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>/opt/moudle/hadoop-3.3.4/data/dfs/namenode</value>
    </property>
<!-- DataNode 数据存储目录 -->
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>/opt/moudle/hadoop-3.3.4/data/dfs/datanode</value>
        </property>
<!-- JournalNode 数据存储目录 -->
        <property>
                <name>dfs.journalnode.edits.dir</name>
                <value>/opt/moudle/hadoop-3.3.4/data/dfs/journaldata</value>
        </property>
<!-- 完全分布式集群名称 -->
        <property>
                <name>dfs.nameservices</name>
                <value>hdpcluster</value>
        </property>
<!-- 集群中 NameNode 节点都有哪些nn1,nn2,nn3 -->
        <property>
                <name>dfs.ha.namenodes.hdpcluster</name>
                <value>nn1,nn2</value>
        </property>
<!-- NameNode 的 RPC 通信地址 -->
        <property>
                <name>dfs.namenode.rpc-address.hdpcluster.nn1</name>
                <value>master:9000</value>
        </property>
        <property>
                <name>dfs.namenode.rpc-address.hdpcluster.nn2</name>
                <value>salve1:9000</value>
        </property>
<!-- NameNode 的 http 通信地址 -->
    <property>
        <name>dfs.namenode.http-address.hdpcluster.nn1</name>
        <value>master:9870</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.hdpcluster.nn2</name>
        <value>salve1:9870</value>
    </property>
<!-- 指定 NameNode 元数据在 JournalNode 上的存放位置 -->
        <property>
                <name>dfs.namenode.shared.edits.dir</name>
                <value>qjournal://master:8485;salve1:8485;salve2:8485/hdpcluster</value>
        </property>
<!-- 访问代理类：client 用于确定哪个 NameNode 为 Active -->
        <property>
                <name>dfs.client.failover.proxy.provider.hdpcluster</name>
                <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
        </property>
<!-- 配置隔离机制，即同一时刻只能有一台服务器对外响应 -->
        <property>
                <name>dfs.ha.fencing.methods</name>
                <value>sshfence
                shell(/bin/true)</value>
        </property>
<!-- 使用隔离机制时需要 ssh 秘钥登录-->
    <property>
       <name>dfs.ha.fencing.ssh.private-key-files</name>
       <value>/home/appview/.ssh/id_rsa</value>
    </property>
<!-- 启动 NN 故障自动切换-->
        <property>
                <name>dfs.ha.automatic-failover.enabled</name>
                <value>true</value>
        </property>
<!-- 指定副本数量 -->
        <property>
                <name>dfs.replication</name>
                <value>3</value>
        </property>
<!-- yarn容器允许分配的最大最小内存 -->
<property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>512</value>
</property>
<property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>4096</value>
</property>
<!-- yarn容器允许管理的物理内存大小 -->
 <property>
     <name>yarn.nodemanager.resource.memory-mb</name>
     <value>4096</value>
 </property>
<!-- 关闭yarn对虚拟内存的限制检查 -->
 <property>
     <name>yarn.nodemanager.vmem-check-enabled</name>
     <value>false</value>
 </property>
</configuration>
3.YARN配置文件
3.1配置yarn-site.xml
[appview@master hadoop]$ vim yarn-site.xml
文件内容如下：
<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at
​
    http://www.apache.org/licenses/LICENSE-2.0
​
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>
​
<!-- Site specific YARN configuration properties -->
<!-- 指定MR走shuffle-->
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
<!-- 启用 resourcemanager ha -->
   <property>
    <name>yarn.resourcemanager.ha.enabled</name>
    <value>true</value>
  </property>
 <!-- 声明两台 resourcemanager 的地址 -->
   <property>
    <name>yarn.resourcemanager.cluster-id</name>
    <value>cluster-yarn</value>
  </property>
<!--指定 resourcemanager 的逻辑列表-->
  <property>
    <name>yarn.resourcemanager.ha.rm-ids</name>
    <value>rm1,rm2</value>
  </property>
<!-- ========== rm1 的配置 ========== -->
<!-- 指定 rm1 的主机名 -->
  <property>
    <name>yarn.resourcemanager.hostname.rm1</name>
    <value>salve1</value>
  </property>
<!-- 指定 rm1 的 web 端地址 -->
  <property>
        <name>yarn.resourcemanager.webapp.address.rm1</name>
        <value>salve1:8088</value>
  </property>
<!-- 指定 rm1 的内部通信地址 -->
  <property>
        <name>yarn.resourcemanager.address.rm1</name>
        <value>salve1:8032</value>
  </property>
<!-- 指定 AM 向 rm1 申请资源的地址 -->
  <property>
        <name>yarn.resourcemanager.scheduler.address.rm1</name>
        <value>salve1:8030</value>
  </property>
<!-- 指定供 NM 连接的地址 -->
  <property>
        <name>yarn.resourcemanager.resource-tracker.address.rm1</name>
        <value>salve1:8031</value>
  </property>
<!-- 对管理员暴露的访问地址管理员通过该地址向RM发送管理命令 -->
  <property>
        <name>yarn.resourcemanager.admin.address.rm1</name>
        <value>salve1:8033</value>
  </property>
 <!-- RM1 configs end -->
<!-- ========== rm2 的配置 ========== -->
<!-- 指定 rm2 的主机名 -->
  <property>
    <name>yarn.resourcemanager.hostname.rm2</name>
    <value>salve2</value>
  </property>
<!-- 指定 rm2 的 web 端地址 -->
  <property>
        <name>yarn.resourcemanager.webapp.address.rm2</name>
        <value>salve2:8088</value>
  </property>
<!-- 指定 rm2 的内部通信地址 -->
  <property>
        <name>yarn.resourcemanager.address.rm2</name>
        <value>salve2:8032</value>
  </property>
<!-- 指定 AM 向 rm2 申请资源的地址 -->
  <property>
        <name>yarn.resourcemanager.scheduler.address.rm2</name>
        <value>salve2:8030</value>
  </property>
<!-- 指定供 NM 连接的地址 -->
  <property>
        <name>yarn.resourcemanager.resource-tracker.address.rm2</name>
        <value>salve2:8031</value>
  </property>
<!-- 对管理员暴露的访问地址管理员通过该地址向RM发送管理命令 -->
  <property>
        <name>yarn.resourcemanager.admin.address.rm2</name>
        <value>salve2:8033</value>
  </property>
<!-- RM2 configs end -->
<!-- ========== rm3 的配置 ========== -->
 <!-- 指定 zookeeper 集群的地址 -->
 <property>
    <name>yarn.resourcemanager.zk-address</name>
    <value>master:2181,salve1:2181,salve2:2181</value>
 </property>
<!-- ha状态切换为自动切换 -->
  <property>
    <name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
    <value>true</value>
  </property>
 <!-- 指定 resourcemanager 的状态信息存储在 zookeeper 集群 -->
 <property>
   <name>yarn.resourcemanager.store.class</name>
   <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
 </property>
<!-- 环境变量的继承 -->
 <property>
 <name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
 </property>
 <!-- 开启日志聚集功能 -->
<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
</property>
<!-- 设置日志聚集服务器地址 -->
<property>
    <name>yarn.log.server.url</name>
    <value>http://master:19888/jobhistory/logs</value>
</property>
<!-- 设置日志保留时间为7天 -->
<property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>604800</value>
</property>
<!--是否启动一个线程检查每个任务正使用的物理内存量，如果任务超出分配值，则直接将其杀掉，默认是 true -->
<property>
 <name>yarn.nodemanager.pmem-check-enabled</name>
 <value>false</value>
</property>
<!--是否启动一个线程检查每个任务正使用的虚拟内存量，如果任务超出分配值，则直接将其杀掉，默认是 true -->
<property>
 <name>yarn.nodemanager.vmem-check-enabled</name>
 <value>false</value>
</property>
</configuration>
4.MapReduce配置文件
4.1配置mapred-site.xml
[appview@master hadoop]$ vim mapred-site.xml
文件内容如下：
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at
​
    http://www.apache.org/licenses/LICENSE-2.0
​
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
​
<!-- Put site-specific property overrides in this file. -->
​
<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
        <property>
        <name>mapreduce.jobhistory.address</name>
                <value>master:10020</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>master:19888</value>
       </property>
</configuration>
5.配置workers
[appview@master hadoop]$ vim  workers

添加内容如下

salve2
salve3
salve4
6.配置env.sh

给以下三个文件添加java路径

[appview@master ~]$ cd $HADOOP_HOME/etc/hadoop
[appview@master hadoop]$ pwd
/opt/moudle/hadoop-3.3.4/etc/hadoop
[appview@master hadoop]$ vim hadoop-env.sh
[appview@master hadoop]$ vim yarn-env.sh
[appview@master hadoop]$ vim mapred-env.sh

文件内容如下

export JAVA_HOME=/opt/moudle/openjdk-8u272

五、分发Hadoop并验证

[appview@master hadoop]$ xsync /opt/moudle/hadoop-3.3.4
[appview@master hadoop]$ hadoop version
Hadoop 3.3.4
Source code repository https://github.com/apache/hadoop.git -r a585a73c3e02ac62350c136643a5e7f6095a3dbb
Compiled by stevel on 2022-07-29T12:32Z
Compiled with protoc 3.7.1
From source with checksum fb9dd8918a7b8a5b430d61af858f6ec
This command was run using /opt/moudle/hadoop-3.3.4/share/hadoop/common/hadoop-common-3.3.4.jar
[appview@master hadoop]$

六、集群启动初始化

1.先启动journalnode在master、slave1、slave2
[appview@master ~]$ xcall hdfs --daemon start journalnode
​
##### 查看进程
[appview@master ~]$ xcall jps
----------- master -----------------
27856 Jps
22881 QuorumPeerMain
23469 JournalNode
----------- salve1 -----------------
24288 Jps
19649 QuorumPeerMain
19906 JournalNode
----------- salve2 -----------------
12465 QuorumPeerMain
12679 JournalNode
16397 Jps
2.namenode格式化并启动namenode查看进程
[appview@master ~]$ hdfs namenode -format
[appview@master ~]$ hdfs --daemon start namenode
[appview@master ~]$ jps
----------- master -----------------
27856 Jps
22881 QuorumPeerMain
23469 JournalNode
23161 NameNode
3.在slave1同步namenode信息
[appview@salve1 ~]$ hdfs namenode -bootstrapStandby
4.格式化ZKFC
[appview@master ~]$ hdfs zkfc -formatZK
5.停止所有的Hadoop的进程，然后重新启动
#### 停止hdfs进程
[appview@master ~]$ stop-dfs.sh
​
### 开启hdfs进程
[appview@master ~]$ start-dfs.sh
​
#### 切换到安装有yarn的salve1或者salve2启动yarn进程
[appview@salve1 ~]$ start-yarn.sh

七、查看所有进程

[appview@master ~]$ xcall jps
----------- master -----------------
22881 QuorumPeerMain
23670 DFSZKFailoverController
29142 Jps
23161 NameNode
23469 JournalNode
23902 JobHistoryServer
----------- salve1 -----------------
25600 Jps
19649 QuorumPeerMain
19906 JournalNode
20020 DFSZKFailoverController
20174 ResourceManager
19775 NameNode
----------- salve2 -----------------
12465 QuorumPeerMain
12580 DataNode
13220 NodeManager
12679 JournalNode
17400 Jps
13098 ResourceManager
----------- salve3 -----------------
3057 Jps
1972 DataNode
2103 NodeManager
----------- salve4 -----------------
2088 NodeManager
3002 Jps
1964 DataNode
[appview@master ~]$

八、Web端查看HDFS的Web页面

遵从一主一备实现高可用，以上的配置文件以及包含了zookeeper的高可用了
HDFS的Web页面1：http://master:9870
HDFS的Web页面2：http://salve1:9870
YARN的Web页面：http://salve1:8088/cluster
标签： 大数据 hadoop debian
本文转载自: https://blog.csdn.net/2301_77356579/article/details/138423204
版权归原作者 ShoShin1020 所有， 如有侵权，请联系我们删除。
登录后发布评论

“大数据技术之Hadoop-3.3.4高可用安装部署操作文档”的评论:

还没有评论
关于作者
...
overfit同步小助手
文章同步
相关阅读

【保姆级教程】实操 Linux 磁盘管理：硬盘选型 & 分区挂载

通过visual studio进行dump文件调试和分析

开源低代码平台-Microi吾码-V8函数列表-后端

rabbitMQ

web渗透测试漏洞复现：Springboot Actuator未授权漏洞复现

如何给SaaS应用做安全

【软件测试】自动化常用函数

文章导航