0


【大数据】Hadoop集群搭建(8249字)

文章目录

因上努力

个人主页:丷从心

系列专栏:大数据

学习指南:【进收藏夹吃灰系列】大数据学习指南

果上随缘


NAT配置

  • 编辑 → \rightarrow →虚拟网络编辑器 → \rightarrow →移除 V M n e t 8 → VMnet8 \rightarrow VMnet8→添加 V M n e t 8 → N A T VMnet8 \rightarrow NAT VMnet8→NAT模式

IP配置

DEVICE=eth0
BOOTPROTO=static
NM_CONTROLLED=yes
ONBOOT=yes
TYPE=Ethernet
IPADDR=192.168.123.10
NETMASK=255.255.255.0
GATEWAY=192.168.123.2
DNS1=202.106.0.20
HWADDR=00:0c:29:1e:d0:84

SecureCRT配置

Properties
Terminal
Emulation
  •                                     T                            e                            r                            m                            i                            n                            a                            l                            →                            L                            i                            n                            u                            x                                  Terminal \rightarrow Linux                     Terminal→Linux
    
  •                                     S                            c                            r                            o                            l                            l                            b                            a                            c                            k                                                         b                            u                            f                            f                            e                            r                            →                            128000                                  Scrollback \ buffer \rightarrow 128000                     Scrollback buffer→128000
    
Appearance
  •                                     C                            u                            r                            r                            e                            n                            t                                                         c                            o                            l                            o                            r                                                         s                            c                            h                            e                            m                            e                            →                            T                            r                            a                            d                            i                            t                            i                            o                            n                            a                            l                                  Current \ color \ scheme \rightarrow Traditional                     Current color scheme→Traditional
    
  •                                     C                            h                            a                            r                            a                            c                            t                            e                            r                                                         e                            n                            c                            o                            d                            i                            n                            g                            →                            U                            T                            F                            −                            8                                  Character \ encoding \rightarrow UTF-8                     Character encoding→UTF−8
    

Java安装

环境变量配置
vim ~/.bashrc
exportJAVA_HOME=/usr/local/src/jdk_1.8.0_111
exportCLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
exportPATH=$PATH:$JAVA_HOME/bin

Hadoop 1.0安装

建立临时文件夹
mkdir tmp
修改配置文件
masters
vim masters
master
slaves
vim slaves
slave1
slave2
core-site.xml
vim core-site.xml
<configuration><property><name>hadoop.tmp.dir</name><value>/usr/local/src/hadoop_1.2.1/tmp</value></property><property><name>fs.default.name</name><value>hdfs://192.168.88.10:9000</value></property></configuration>
mapred-site.xml
vim mapred-site.xml
<configuration><property><name>mapred.job.tracker</name><value>http://192.168.88.10:9001</value></property></configuration>
hdfs-site.xml
vim hdfs-site.xml
<configuration><property><name>dfs.replication</name><value>3</value></property></configuration>
hadoop-env.sh
vim hadoop-env.sh
exportJAVA_HOME=/usr/local/src/jdk_1.6.0_45
环境变量配置
vim ~/.bashrc
exportHADOOP_HOME=/usr/local/src/hadoop_1.2.1
exportPATH=$PATH:$HADOOP_HOME/bin

Hadoop 2.0安装

修改配置文件
hadoop-env.sh
vim hadoop-env.sh
exportJAVA_HOME=${JAVA_HOME}
yarn-env.sh
vim yarn-env.sh
exportJAVA_HOME=${JAVA_HOME}
slaves
vim slaves
slave1
slave2
core-site.xml
vim core-site.xml
<configuration><property><name>fs.defaultFS</name><value>hdfs://master:9000</value><description>指定默认的文件系统</description></property><property><name>hadoop.tmp.dir</name><value>file:/usr/local/src/hadoop_2.6.1/tmp</value><description>Hadoop 临时文件存放目录</description></property></configuration>
hdfs-site.xml
vim hdfs-site.xml
<configuration><property><name>dfs.namenode.secondary.http-address</name><value>master:9001</value><description>SecondaryNameNode IP 地址和端口</description></property><property><name>dfs.namenode.name.dir</name><value>file:/usr/local/src/hadoop_2.6.1/dfs/name</value><description>fsimage 镜像存放目录</description></property><property><name>dfs.datanode.data.dir</name><value>file:/usr/local/src/hadoop_2.6.1/dfs/data</value><description>HDFS 数据文件存放目录</description></property><property><name>dfs.replication</name><value>3</value><description>Block 副本数</description></property></configuration>
mapred-site.xml
cp mapred-site.xml.template mapred-site.xml
vim mapred-site.xml
<configuration><property><name>mapreduce.framework.name</name><value>yarn</value><description>指定运行 MapReduce 的环境是 Yarn</description></property><property><name>mapreduce.map.memory.mb</name><value>1536</value><description>调度器为每个 Map Task 申请的内存数, 各 Job 也可以单独指定, 如果实际使用的资源量超过该值, 则会被强制杀死</description></property><property><name>mapreduce.map.java.opts</name><value>-Xmx1024M</value></property><property><name>mapreduce.reduce.memory.mb</name><value>3072</value><description>调度器为每个 Reduce Task 申请的内存数, 各 Job 也可以单独指定, 如果实际使用的资源量超过该值, 则会被强制杀死</description></property><property><name>mapreduce.reduce.java.opts</name><value>-Xmx2560M</value></property><property><name>mapreduce.job.reduce.slowstart.completedmaps</name><value>0.1</value><description>当 Map Task 完成 10% 时, 为 Reduce Task 申请资源, Reduce Task 开始进行拷贝 Map Task 结果数据和执行 Reduce Shuffle 操作, 默认值为 0.05</description></property><!--Hadoop 历史服务器--><property><name>mapreduce.jobhistory.address</name><value>master:10020</value><description>MapReduce JobHistory 服务器进程间通信地址</description></property><property><name>mapreduce.jobhistory.webapp.address</name><value>master:19888</value><description>MapReduce JobHistory 服务器的 Web UI</description></property><property><name>mapreduce.jobhistory.done-dir</name><value>/mr-history/done</value><description>已执行完毕的作业信息存放目录</description></property><property><name>mapreduce.jobhistory.intermediate-done-dir</name><value>/mr-history/tmp</value><description>正在运行的作业信息存放目录</description></property><property><name>yarn.app.mapreduce.am.staging-dir</name><value>/mr-history/hadoop-yarn</value><description>MapReduce 作业提交时所使用的临时目录</description></property></configuration>
yarn-site.xml
vim yarn-site.xml
<configuration><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value><description>NodeManager 上运行的附属服务</description></property><property><name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name><value>org.apache.hadoop.mapred.ShuffleHandler</value></property><property><name>yarn.resourcemanager.address</name><value>master:8032</value><description>客户端访问 ResourceManager 的地址</description></property><property><name>yarn.resourcemanager.scheduler.address</name><value>master:8030</value><description>ApplicationMaster 访问 ResourceManager 的地址</description></property><property><name>yarn.resourcemanager.resource-tracker.address</name><value>master:8035</value><description>NodeManager 访问 ResourceManager 的地址</description></property><property><name>yarn.resourcemanager.admin.address</name><value>master:8033</value><description>管理员访问 ResourceManager 的地址</description></property><property><name>yarn.resourcemanager.webapp.address</name><value>master:8088</value><description>Web UI 访问 ResourceManager 的地址</description></property><!--关闭内存检查--><property><name>yarn.nodemanager.pmem-check-enabled</name><value>false</value><description>是否启动一个线程检查每个任务正使用的物理内存量, 如果任务超出分配值, 则直接将其杀掉, 默认值为true</description></property><property><name>yarn.nodemanager.vmem-check-enabled</name><value>false</value><description>是否启动一个线程检查每个任务正使用的虚拟内存量, 如果任务超出分配值, 则直接将其杀掉, 默认值为true</description></property><!--日志聚合--><property><name>yarn.log-aggregation-enable</name><value>true</value><description>开启日志聚合功能, 日志保存在 HDFS 上</description></property><property><name>yarn.log-aggregation.retain-seconds</name><value>86400</value><description>聚合后的日志在 HDFS 上的保存时间, 单位为秒</description></property><property><name>yarn.log.server.url</name><value>http://master:19888/jobhistory/logs</value><description>日志聚合服务器 URL</description></property></configuration>
环境变量配置
vim ~/.bashrc
exportHADOOP_HOME=/usr/local/src/hadoop_2.6.1
exportPATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

IP与主机名映射关系配置

hostname配置
  •                                     h                            o                            s                            t                            n                            a                            m                            e                                  hostname                     hostname临时生效
    
hostname master
  •                                     h                            o                            s                            t                            n                            a                            m                            e                                  hostname                     hostname永久生效
    
vim /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=master
映射关系配置
vim /etc/hosts
192.168.123.10 master
192.168.123.11 slave1
192.168.123.12 slave2
Windows hosts配置
  • 路径:C:\Windows\System32\drivers\etc\hosts
192.168.123.10 master
192.168.123.11 slave1
192.168.123.12 slave2

关闭防火墙

防火墙临时关闭
systemctl stop firewalld.service
防火墙永久关闭
systemctl disable firewalld.service

关闭SELinux

SELinux临时关闭
setenforce 0
SELinux永久关闭
vim /etc/selinux/config
SELINUX=disabled

建立互信关系

ssh-keygen -t rsa
cd ~/.ssh
将公钥文件id_rsa.pub拷贝到authorized_keys
cat id_rsa.pub > authorized_keys
修改权限
chmod600 authorized_keys
将slave_1、slave_2的公钥拷贝到master的authorized_keys
ssh slave1 cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
远程拷贝
scp-rp /root/.ssh/authorized_keys root@slave1:/root/.ssh/authorized_keys
测试互信关系
ssh slave1

Hadoop启动

NameNode格式化
  • 第一次启动 H a d o o p Hadoop Hadoop需要对 N a m e N o d e NameNode NameNode进行格式化
hadoop namenode -format
报错解决
2734 -- main class information unavailable
rm-rf /tmp/hsperfdata_root/

标签: 大数据 Hadoop

本文转载自: https://blog.csdn.net/from__2024_06_01/article/details/139409001
版权归原作者 丷从心 所有, 如有侵权,请联系我们删除。

“【大数据】Hadoop集群搭建(8249字)”的评论:

还没有评论