windows下配置hadoop3.1.4环境

文章目录

说明

目前情况

现在在做一个hadoop集群，本机是windows11环境，hadoop集群是centos环境，已经搭建好了一个四节点的集群，分别是master、slave1、slave2、slave3,分别在四台虚拟机上。现在对集群的文件进行下载操作遇到报错没有Hadoop环境。于是进行环境配置

检查Java环境

运行命令行

java -version

配置Hadoop环境

下载解压hadoop3.1.4

下载插件

winutils 是 Hadoop 在 Windows 操作系统上运行时所需的一个工具集。它主要用于提供一些 Unix/Linux 系统上默认存在的功能和命令的替代实现，因为 Hadoop 大部分是在这些系统上开发和运行的

下载地址 链接: link

需要这两个文件，放入hadoop的bin文件夹下

设置环境变量

报错

设置完环境变量，命令行输入hadoop version发现报错

hadoop version

报错

解决报错，原因是Java_Home目录中含有空格，hadoop识别不了空格

具体原因可能是 在许多编程环境和脚本中，空格被用作参数分隔符。当路径中包含空格时，系统可能会将其误解为多个参数，而不是一个完整的路径。例如，路径 C:\Program Files\Java\jdk 会被解析为三个不同的部分。

查看我们的java环境

换个位置，修改环境

命令行再次输入

到这基本能用了，解决了开头说的下载操作，想在window下启动可以继续往下看

修改Hadoop配置文件

修改的文件都在这个目录下面

修改core-site.xml文件

<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!--Licensed under the ApacheLicense,Version2.0(the "License");
  you may not use this file except in compliance withtheLicense.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed toin writing, software
  distributed under the License is distributed on an "AS IS"BASIS,WITHOUTWARRANTIESORCONDITIONSOFANYKIND, either express or implied.
  See the Licensefor the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.--><!--Put site-specific property overrides in this file.--><configuration><property><name>fs.defaultFS</name><value>hdfs://localhost:9000</value></property></configuration>

修改hdfs-site.xml文件

<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!--Licensed under the ApacheLicense,Version2.0(the "License");
  you may not use this file except in compliance withtheLicense.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed toin writing, software
  distributed under the License is distributed on an "AS IS"BASIS,WITHOUTWARRANTIESORCONDITIONSOFANYKIND, either express or implied.
  See the Licensefor the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.--><!--Put site-specific property overrides in this file.--><configuration><property><name>dfs.replication</name><value>1</value></property><property><name>dfs.namenode.name.dir</name><value>file:///C:/hadoop/hadoop-3.1.4/data/namenode</value></property><property><name>dfs.datanode.data.dir</name><value>file:///C:/hadoop/hadoop-3.1.4/data/datanode</value></property></configuration>

这里会存在问题

1 需要自己在hadoop下新建文件夹data（可以换成其他名字），在新datanode和namenode文件夹

2 注意路径写法，hadoop不能读取windows下的路径，我们需要把原本的路径换成URI格式

修改yarn-site.xml文件

<?xml version="1.0"?><!--Licensed under the ApacheLicense,Version2.0(the "License");
  you may not use this file except in compliance withtheLicense.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed toin writing, software
  distributed under the License is distributed on an "AS IS"BASIS,WITHOUTWARRANTIESORCONDITIONSOFANYKIND, either express or implied.
  See the Licensefor the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.--><configuration><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><property><name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name><value>org.apache.hadoop.mapred.ShuffleHandler</value></property></configuration>

修改mapred-site.xml文件

<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!--Licensed under the ApacheLicense,Version2.0(the "License");
  you may not use this file except in compliance withtheLicense.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed toin writing, software
  distributed under the License is distributed on an "AS IS"BASIS,WITHOUTWARRANTIESORCONDITIONSOFANYKIND, either express or implied.
  See the Licensefor the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.--><!--Put site-specific property overrides in this file.--><configuration><property><name>mapreduce.framework.name</name><value>yarn</value></property></configuration>

修改hadoop-env.sh文件

修改hadoop-env.cmd文件

格式化

命令行进入bin目录，执行格式化命令

hdfs namenode –format

启动集群

命令行进入sbin目录运行start-all脚本

.\start-all.cmd

执行完命令弹出4个窗口

查看第一个窗口有无报错

没有报错基本就运行成功了

我这里jps没有输出，不知道什么原因，求解

游览器访问

打开游览器访问localhost:9870