Windows11编译Hadoop3.3.6源码

由于

https://github.com/kontext-tech/winutils

还未发布3.3.6版本，因此尝试源码编译。

本文 hadoop-3.3.6 编译后 winutils 等文件获取

GitHub 地址

目录

环境和安装包准备，见2

• Git-2.43.0-64-bit.exe
• jdk-8u381-windows-x64.exe
• node-v21.2.0-x64.msi
• apache-ant-1.10.14-bin.zip
• apache-maven-3.9.5-bin.zip
• protoc-25.1-win64.zip
• cmake-3.28.0-rc5-windows-x86_64.zip
• GetGnuWin32-0.6.3.exet
• zlib.tar.gz
• Visual Studio 2010 Professional
• hadoop-3.3.6-src.tar.gz

zlib编译

方法一：

• VS2010打开zlib路径\contrib\vstudio\vc10\zlibvc.sln

zlib的vs sln解决方案文件包含6个工程。其中：
(1). zlibvc生成动态链接库zlibwapi.dll和对应的导入库zlibwapi.lib
(2). zlibstat生成静态链接库zlibstat.lib
(3). testzlibdll测试生成的动态链接库zlibwapi.dll
(4). testzlib测试生成的静态链接库zlibstat.lib

• debug模式下编译zlibvc和zlibstat，在zlib路径\contrib\vstudio\vc10\x64\的ZlibStatDebug和ZlibDllDebug文件夹中获取编译生成的文件
• 将这三个文件存入单独的文件夹中，将路径加入系统环境变量path

方法二：

VS命令行工具进入到zlib根目录，执行

nmake -f win32/Makefile.msc

,在根目录下生成：zlib.lib(静态库) zdll.lib(动态库的导入库) zlib1.dll(动态库)

配置文件更改

1. maven阿里云镜像

修改

apache-maven-3.9.5路径\conf\settings.xml

文件镜像mirrors节点，使用阿里云仓库，添加如下代码。

<mirror><id>aliyunmaven</id><mirrorOf>*</mirrorOf><name>阿里云公共仓库</name><url>https://maven.aliyun.com/repository/public</url></mirror>

不更改会很慢，亲测！！！

2. Node版本

• 编辑hadoop-3.3.6-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-catalog\hadoop-yarn-applications-catalog-webapp的pom.xml文件
• findnodeVersion，将<nodeVersion>${nodejs.version}</nodeVersion>改为自己的node版本<nodeVersion>v21.2.0</nodeVersion>
• 如果不更改会出现错误如下。

[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.11.2:yarn (yarn install) on project 
hadoop-yarn-applications-catalog-webapp: Failed to run task: 'yarn ' failed. 
org.apache.commons.exec.
ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1]

3. 越过Javadoc检查

1. 在pom.xml中禁用严格检查<additionalparam>-Xdoclint:none</additionalparam>

<plugins>
  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-javadoc-plugin</artifactId>
    <configuration>
      <additionalparam>-Xdoclint:none</additionalparam>
    </configuration>
  </plugin>
</plugins>

特别是

hadoop-3.3.6-src\hadoop-common-project\hadoop-annotations\pom.xml

需要更改，否则会出现如下错误。

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:3.0.1:jar (module-javadocs) on project hadoop-annotations: MavenReportException: Error while generating Javadoc:

1. mvn命令构建时跳过 Javadocmvn 其他参数 -Dmaven.javadoc.skip=true verify

编译Hadoop

• VS命令行工具cd Hadoop文件夹下
• 执行命令mvn clean package -Pdist,native-win -DskipTests -Dtar -Dmaven.javadoc.skip=true
• 编译成功
• 编译后文件目录为hadoop-3.3.6-src\hadoop-dist\target

Error,其他报错见3

1. RelWithDebInfo does not exist

hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\bin\RelWithDebInfo does not exist

错误是这个目录不存在，则在

hadoop-hdfs-project\hadoop-hdfs-native-client\target\native

下手动创建这个目录，并根据提示在重新执行命令时添加参数

-rf :hadoop-hdfs-native-client

，如下。

mvn package -Pdist,native-win -DskipTests -Dtar -Dmaven.javadoc.skip=true -rf :hadoop-hdfs-native-client

1. hadoop-hdfs-native-client

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.x:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec failonerror="true" dir="hadoop路径\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native" executable="cmake">

修改

hadoop-hdfs-project\hadoop-hdfs-native-client\pom.xml

文件

failonerror

的值true为false

3. convert-ms-winutils

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (convert-ms-winutils) on project hadoop-common: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (convert-ms-winutils) on project hadoop-common: Command execution failed.`

• 添加环境变量 C:\Windows\Microsoft.NET\Framework64\v4.0.30319
• 或修改冲突文件名，确定引用 C:\Windows\Microsoft.NET\Framework64\v4.0.30319文件

相关文章：

1. windows上编译hadoop
2. Windows10 Hadoop3.3.0源码编译
3. Hadoop 3.2.1 win10 64位系统 vs2015 编译
4. Hadoop源码编译打包
5. hadoop源码编译
6. Windows 7 64bit和Visual Studio 2010下安装及使用Zlib 1.2.5
7. VS2010 编译 Zlib库-1.2.11（zip解压库）
8. Windows中zlib的安装与配置
9. Unable to build Maven project due to Javadoc error?
10. maven项目install时候一直报错，检查Maven-javadoc-plugin声明错误（Java 8 配置Maven-javadoc-plugin）或者警告: @throws没有说明
11. Windows7-64编译hadoop-3.2.0
12. Hadoop on Windows Building/ Installation Error