0


CENTOS上的网络安全工具(二十四)Windows下的Hadoop+Spark编程环境构建

    前面我们搭建了hadoop集群,spark集群,也利用容器构建了spark的编程环境。但是一般来说,就并行计算程序的开发,一刚开始一般是在单机上的,比如hadoop的single node。但是老师弄个容器或虚拟机用vscode远程访问式开发,终究还是有些不爽。还好,hadoop和spark都是支持windows的。不妨,我们弄个windows下的开发环境。

    然而,windows下开发环境的构建,需要一个转换程序winutils.exe,这个需要根据下载的hadoop的版本对应编译。而且,编译好的exe文件在网上并不好找,一些大虾们编译完了,往往挂在csdn上还要收点费……。所以,还是自己编译吧,好歹还能学点东西。

一、预备环境构建

    很多网上的攻略仅仅提到了需要安装些什么软件,但并没有提及为什么。所以,我在安装的时候,本着剃刀原则,尽量少装,然后摔一个坑多装一个。所以,也基本搞清了这些工具的作用——实际上就是,maven集成了一堆的跨平台编译器,并通过插件来调用它们。如果调用的时候找不到,自然就会报错。

    所欲i,这里提到的软件,有一个算一个,都得装。

1. CMAKE

    到官方网站Download | CMake下载安装包进行安装: 

     如果不打算手工修改环境变量的话,记得勾上这个:

     成功安装,则可在命令行执行版本检查:
Microsoft Windows [版本 10.0.19044.1288]
(c) Microsoft Corporation。保留所有权利。

C:\Users\pig>cmake -version
cmake version 3.26.4

CMake suite maintained and supported by Kitware (kitware.com/cmake).

C:\Users\pig>

2. JDK 1.8

    不论是maven的配置文件,还是hadoop的pom文件,默认的java版本都是JDK 1.8。虽然我们之前落过1.8的坑,换成了11,但这里还是改回来的好。否则,不知哪个犄角旮旯的1.8没有改成11,都有可能造成编译失败。

    下载后安装:

    安装成功后,应该能够在cmd中查看版本号:
Microsoft Windows [版本 10.0.19044.1288]
(c) Microsoft Corporation。保留所有权利。

C:\Users\pig>java -version
java version "1.8.0_202"
Java(TM) SE Runtime Environment (build 1.8.0_202-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.202-b08, mixed mode)

C:\Users\pig>
    然后在高级系统设置中设置环境变量JAVA_HOME,指向刚才的安装目录。这里我们的安装目录更改过,默认应该在program files目录下面。

3. Maven

    在Maven的官方网站Maven – Download Apache Maven下载编译好的binary zip版本,直接解压到windows里。![](https://img-blog.csdnimg.cn/bb4a91040ae945ab9713f6f9974357f6.png)

(1)设置环境变量

    maven有2个环境变量需要设置,一个是PATH,一个是MAVEN_HOME

    MAVEN_HOME需要设置为我们释放它的目录:

    PATH则是需要将maven目录下的bin的绝对目录添加进去:

     设置完成后,重启一个cmd,输入mvn -v命令,应该能看到如下结果:
C:\Users\pig>mvn -v
Apache Maven 3.9.2 (c9616018c7a021c1c39be70fb2843d6f5f9b8a1c)
Maven home: C:\maven
Java version: 1.8.0_202, vendor: Oracle Corporation, runtime: C:\Java\jre
Default locale: zh_CN, platform encoding: GBK
OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"

C:\Users\pig>

(2)设置本地仓库及阿里镜像

    在MAVEN_HOME/conf目录下的settings.xml文件中,如下设置:

  **  本地仓库目录为c:/m2repo,**对应新建这个目录。默认这个目录应该在当前用户目录下的.m2目录中,方便起见,改一下。
<!-- localRepository
   | The path to the local repository maven will use to store artifacts.
   |
   | Default: ${user.home}/.m2/repository
  <localRepository>/path/to/local/repo</localRepository>
  -->
  <localRepository>c:/m2repo</localRepository>

** 添加阿里镜像,**这样下载能够快不少:

  <mirrors>
    <!-- mirror
     | Specifies a repository mirror site to use instead of a given repository. The repository that
     | this mirror serves has an ID that matches the mirrorOf element of this mirror. IDs are used
     | for inheritance and direct lookup purposes, and must be unique across the set of mirrors.
     |
    <mirror>
      <id>mirrorId</id>
      <mirrorOf>repositoryId</mirrorOf>
      <name>Human Readable Name for this Mirror.</name>
      <url>http://my.repository.com/repo/path</url>
    </mirror>
     -->
    <mirror>
      <id>aliyunmaven</id>
      <mirrorOf>*</mirrorOf>
      <name>阿里云公共仓库</name>
      <url>https://maven.aliyun.com/repository/public</url>
    </mirror> 
    <mirror>
      <id>maven-default-http-blocker</id>
      <mirrorOf>external:http:*</mirrorOf>
      <name>Pseudo repository to mirror external repositories initially using HTTP.</name>
      <url>http://0.0.0.0/</url>
      <blocked>true</blocked>
    </mirror>
  </mirrors>
    配置好后,新开cmd窗口,执行 mvn help:system命令
C:\Users\pig>mvn help:system
[INFO] Scanning for projects...
Downloading from aliyunmaven: https://maven.aliyun.com/repository/public/org/apache/maven/plugins/maven-clean-plugin/3.2.0/maven-clean-plugin-3.2.0.pom
…………
…………
MAVEN_PROJECTBASEDIR=C:\Users\pig

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  13.828 s
[INFO] Finished at: 2023-06-10T21:02:29+08:00
[INFO] ------------------------------------------------------------------------
[WARNING]
[WARNING] Plugin validation issues were detected in 1 plugin(s)
[WARNING]
[WARNING]  * org.apache.maven.plugins:maven-site-plugin:3.12.1
[WARNING]
[WARNING] For more or less details, use 'maven.plugin.validation' property with one of the values (case insensitive): [BRIEF, DEFAULT, VERBOSE]
[WARNING]
    顺利的话,mvn会下载一些默认的依赖项,此时指定的本地库目录下应该就不为空了:

4. 安装Visual Studio

     如果安装这个大家伙的唯一目的是编译protobuf获得protoc.exe文件。如果你决定下载已编译好的可执行文件用的话(也就是不编译protobuf),完全可以选择避开这玩意。如果要编译,那就装2010以上的版本。最好2010,因为上到2019,还会出现一些别的问题。

  **  然而,如果还打算继续编译hadoop,并且不希望看到诸如“ Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec”这样的错误的话,VS还是得装的,毕竟maven可能会通过exec插件来调用vs的编译器来进行编译。**

** 5. 安装Git**

    安装git,除了可能会用到git clone或者git submodule update以外,最主要的作用,是使用Git中自带的git bash。因为后面在编译的时候,maven会使用ant插件运行sh脚本,这个需要bash支持。 

    下载的官网地址在Git - Downloading Package (git-scm.com)        ![](https://img-blog.csdnimg.cn/1a446a119de049039ba967fbaca69035.png)

     一路默认安装就好,但是默认条件下,加入环境变量PATH的是git-cmd目录,bash并不在这个目录中,

     这也是我最后编译成功的时候的PATH环境变量情况。可以忽略上面那个ant,保险起见我也手工下载安装了一个,但是感觉没啥作用。maven的antrun插件应该取代它的作用了。

二、编译protobuf

    之所以编译protobuf,是因为hadoop编译中需要protoc.exe这个编译工具。所以如果不想麻烦的话,直接去下载编译好的exe文件也行:https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protoc-2.5.0-win32.zip。

    但是不管是编译源码还是直接下载可执行文件,**都一定要认准2.5.0版本**——因为hadoop目前的源码编译中要求,不是该版本会造成各种莫名奇妙的插件执行错误。**切记切记!!!**

1. 下载protobuf

    protobuf的下载地址在github上:Release Protocol Buffers v2.5.0 · protocolbuffers/protobuf · GitHub

    因为魔法的缘故,打开这个网站需要碰运气,多刷几次就行:

     两个红圈中的都是源代码,看起来连接不一样,实际下下来的压缩包完全一样。如果是要下可执行文件,中间那个就是,对应上面我们已经给出的链接。

2. 编译生成protoc

     2. 2.5.0版本的protobuf编译起来十分简单,因为它提供了visual studio的工程文件sln,在protobuf\vsprojects目录下,使用VS Studio将其打开即可:        ![](https://img-blog.csdnimg.cn/f7cc1647bbdc4c98b60e4d0d7c0b7428.png)

    使用Visual Studio编译的过程就不赘述了,点几个按钮而已。 当然,如果使用的是VS 2019,需要注意以下这个问题:

     在默认启动项libprotobuf的header文件加中,找到config.h文件,在其中添加一个宏定义
#define _SILENCE_STDEXT_HASH_DEPRECATION_WARNINGS
    以避免因为版本问题出现hash_map错误

    并且,为了代码能够运行得快点,应该打开各项目的属性面板,选择release 和win32版本:

另外 ,为了避免编译顺序引起的不知名错误,建议一个一个项目的单独编译,不要一上来就直接编译整个工程。总之我这么干冒出来好几百个错误,如果一个一个来,就很顺利通过了。

3. latest protobuf

    虽然用不着,但既然我之前掉坑里了(直接选用了23版本的最新protobuf),这里还是说几句。最新版的protobuf没有提供vsprojects目录和sln文件,需要通过命令行生成;当然也可以使用Ninja进行编译。所有这些的详细细节,都在protobuf\cmake\目录下的Readme.rd文件中。

(1)安装git,并使用git更新子模块

    其实这一步基本用不上,想用也不好用,所以我就没有把它写在预备条件章节。

官方的说明是,如果我们是使用git clone获得的源码,则需要在protobuf目录下执行更新命令更新子模块,如果是直接下载的zip包或者tar包则不需要。虽然下一步也确实遭遇子模块不齐整的问题,但可以通过直接下载方法解决,这里就不赘述。

C:\protobuf> git submodule update --init --recursive

(2)生成Visual Sutdio工程文件

    根据官方的说明,需要在有CMakeList.txt文件的目录下执行cmake(也就是根目录下):
PS F:\build\protobuf232> cmake -G "Visual Studio 16 2019" -A Win32 -B ./build -DCMAKE_INSTALL_PREFIX=./build/output -DCMAKE_CXX_STANDARD=17
CMake Warning:
  Ignoring extra path from command line:

   "./build/output"

-- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.22621.
--
-- 23.2.0
-- Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR)
CMake Warning at cmake/abseil-cpp.cmake:28 (message):
  protobuf_ABSL_PROVIDER is "module" but ABSL_ROOT_DIR is wrong
Call Stack (most recent call first):
  CMakeLists.txt:339 (include)

CMake Error at third_party/utf8_range/CMakeLists.txt:31 (add_subdirectory):
  The source directory

    F:/build/protobuf232/third_party/abseil-cpp

  does not contain a CMakeLists.txt file.

-- Configuring incomplete, errors occurred!
    然而,执行结果是缺少abseil-cpp第三方文件——提示中对应的那个目录是空的。并且,cmake会提示说要不要通过git submodule update来更新这个第三方插件……因为有魔法的缘故,这件事是比较玄幻的……我曾经git clone成功下载protobuf23.2,也曾经通过git update成功下载abseil-cpp;但重现这一过程的时候,这两样都无论如何不行了,即使用魔法对抗魔法也没有作用。

    PS:

    如果还提示说googletest不齐整,不要犹豫,直接使用protobuf_BUILD_TESTS=OFF干掉之:
PS F:\build\protobuf232> cmake -G "Visual Studio 16 2019" -A Win32 -B ./build -DCMAKE_INSTALL_PREFIX=./build/output -DCMAKE_CXX_STANDARD=17 -Dprotobuf_BUILD_TESTS=OFF

** 下载并补齐abseil-cpp**

    好在这个文件夹是可以在GitHub - abseil/abseil-cpp: Abseil Common Libraries (C++)上下载的:

     对应拷贝进去就行了:        ![](https://img-blog.csdnimg.cn/8d03204b4a6446b0898ff4e2f128f594.png)

     再次执行:
PS F:\build\protobuf232> cmake -G "Visual Studio 16 2019" -A Win32 -B ./build -DCMAKE_INSTALL_PREFIX=./build/output -DCMAKE_CXX_STANDARD=17
CMake Warning:
  Ignoring extra path from command line:

   "./build/output"

-- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.22621.
--
-- 23.2.0
-- Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR)
CMake Warning at third_party/abseil-cpp/CMakeLists.txt:72 (message):
  A future Abseil release will default ABSL_PROPAGATE_CXX_STD to ON for CMake
  3.8 and up.  We recommend enabling this option to ensure your project still
  builds correctly.

-- Performing Test ABSL_INTERNAL_AT_LEAST_CXX17
-- Performing Test ABSL_INTERNAL_AT_LEAST_CXX17 - Success
-- Configuring done (1.9s)
-- Generating done (1.7s)
-- Build files have been written to: F:/build/protobuf232/build
PS F:\build\protobuf232>
    Visual Studio 2019可以用的工程就已经生成了。

(3)编译

    VS2019打开,直接在ALL BUILD上右键生成——请忽略我这个灰色,因为我已经点下去了……

    一阵有惊无险的warning后,好在是成功生成了:

     找到debug底下,这个protoc已经生成了,执行看看版本:
PS F:\build\protobuf232\build\Debug> ./protoc --version
libprotoc 23.2

(4)其它方法

    我第一次成功生成并没有参考官方的方法,而是自己一阵误打误撞进去的cmake+nmake方法。不同之处在于使用cmake生成的nmake文件,而不是sln:
cmake -G “NMake Makefiles” -DCMAKE_BUILD_TYPE=Debug -Dprotobuf_BUILD_TESTS=OFF
    然后在对其nmake && nmake install就行了。具体可参考protobuf windows编译和安装_wx637304bacd051的技术博客_51CTO博客

(5)继续生成java版本的protobuf

     还是以protobuf 23.2版为例(2.5.0版本一样,莫名其妙的问题还少些)。按照官方的说明,将刚生成的protoc.exe文件拷贝到protobuf所在目录的src子目录下。并且在环境变量PATH中设置这个位置,以确保执行时使用的是这个版本。![](https://img-blog.csdnimg.cn/dcc8e8cde310457bbb23d47745526a7a.png)

     网上也有说拷贝到system32下面的。这和windows中cmd寻找命令对应可执行文件的优先序有关,一般来说是先找系统目录,再找环境变量PATH下面,再本级目录(映像如此,准确与否不保证,试试便知)。既然官方说拷贝到src下,应该这个就可以。保险起见,我才加了path。

    然后到java目录下,执行mvn test
F:\build\protobuf232\java>mvn test
[INFO] Scanning for projects...
Downloading from aliyunmaven: https://maven.aliyun.com/repository/public/org/apache/felix/maven-bundle-plugin/3.0.1/maven-bundle-plugin-3.0.1.pom
………………
………………
[WARNING]
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:3.0.0:run (generate-sources) on project protobuf-javalite: An Ant BuildException has occured: The following error occurred while executing this line:
[ERROR] F:\build\protobuf232\java\lite\generate-sources-build.xml:4: Execute failed: java.io.IOException: Cannot run program "F:\build\protobuf232\java\lite\..\..\protoc" (in directory "F:\build\protobuf232\java\lite"): CreateProcess error=2, 系统找不到指定的文 件。
[ERROR] around Ant part ...<ant antfile="generate-sources-build.xml" />... @ 4:49 in F:\build\protobuf232\java\lite\target\antrun\build-main.xml
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
     然后出现如上错误。

    检查本地库,antrun-plugin好好的在那里,那就不是插件的问题

     仔细看ERROR信息,实际的意思就是说再protobuf工作目录的根目录上,没有找到protoc.exe,这个真是……。**把src里面那个在根里再拷贝一份吧。**

    接下来,得到新的错误:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.1:testCompile (default-testCompile) on project protobuf-java: Compilation failure: Compilation failure:
[ERROR] /F:/build/protobuf232/java/core/src/test/java/com/google/protobuf/DescriptorsTest.java:[68,25] 找不到符号
[ERROR]   符号:   类 UnittestRetention
[ERROR]   位置: 程序包 protobuf_unittest
[ERROR] /F:/build/protobuf232/java/core/src/test/java/com/google/protobuf/DescriptorsTest.java:[572,27] 找不到符号
[ERROR]   符号:   变量 UnittestRetention
[ERROR]   位置: 类 com.google.protobuf.DescriptorsTest
[ERROR] /F:/build/protobuf232/java/core/src/test/java/com/google/protobuf/DescriptorsTest.java:[573,37] 找不到符号
[ERROR]   符号:   变量 UnittestRetention
[ERROR]   位置: 类 com.google.protobuf.DescriptorsTest
[ERROR] /F:/build/protobuf232/java/core/src/test/java/com/google/protobuf/DescriptorsTest.java:[574,37] 找不到符号
[ERROR]   符号:   变量 UnittestRetention
[ERROR]   位置: 类 com.google.protobuf.DescriptorsTest
[ERROR] /F:/build/protobuf232/java/core/src/test/java/com/google/protobuf/DescriptorsTest.java:[575,37] 找不到符号
[ERROR]   符号:   变量 UnittestRetention
[ERROR]   位置: 类 com.google.protobuf.DescriptorsTest
     这个错误对应的DescriptorsTest.java源文件在路径protobuf/java/core/src/test/java/com/google/protobuf/下,引起错误的原因是import protobuf_unittest.UnittestRetention找不到,而这个东西对应的目录在F:\build/protobuf232\java\lite\target\test-classes\protobuf_unittest下面。

    问题是在src里面找不到对应的这个类,Google也找不到,不知为何?

** 好在只是在测试代码中,直接把测试代码的编译忽略掉好了。**

** 加上-Dmaven.test.skip=true。**

** 直接install,这样就搞定了**

F:\build\protobuf232\java>mvn install -Dmaven.test.skip=true
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Protocol Buffers [BOM]                                             [pom]
[INFO] Protocol Buffers [Parent]                                          [pom]
[INFO] Protocol Buffers [Lite]                                         [bundle]
[INFO] Protocol Buffers [Core]                                         [bundle]
[INFO] Protocol Buffers [Util]                                         [bundle]
[INFO] Protocol Buffers [Kotlin-Core]                                     [jar]
[INFO] Protocol Buffers [Kotlin-Lite]                                     [jar]
[INFO]
[INFO] ------------------< com.google.protobuf:protobuf-bom >------------------
[INFO] Building Protocol Buffers [BOM] 3.23.2                             [1/7]
[INFO]   from bom\pom.xml
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- install:3.1.0:install (default-install) @ protobuf-bom ---
[INFO] Installing F:\build\protobuf232\java\bom\pom.xml to 
…………
…………
[INFO] Protocol Buffers [Kotlin-Lite] ..................... SUCCESS [  9.266 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:59 min
[INFO] Finished at: 2023-06-11T13:27:16+08:00
[INFO] ------------------------------------------------------------------------
[WARNING]
[WARNING] Plugin validation issues were detected in 9 plugin(s)
[WARNING]
[WARNING]  * org.apache.maven.plugins:maven-jar-plugin:2.6
[WARNING]  * org.apache.maven.plugins:maven-compiler-plugin:3.6.1
[WARNING]  * org.apache.maven.plugins:maven-resources-plugin:3.1.0
[WARNING]  * org.jetbrains.kotlin:kotlin-maven-plugin:1.6.0
[WARNING]  * org.codehaus.mojo:build-helper-maven-plugin:1.10
[WARNING]  * org.codehaus.mojo:animal-sniffer-maven-plugin:1.20
[WARNING]  * org.apache.maven.plugins:maven-antrun-plugin:3.0.0
[WARNING]  * org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5
[WARNING]  * org.apache.felix:maven-bundle-plugin:3.0.1
[WARNING]
[WARNING] For more or less details, use 'maven.plugin.validation' property with one of the values (case insensitive): [BRIEF, DEFAULT, VERBOSE]
[WARNING]

F:\build\protobuf232\java>
     最后package一下:
F:\build\protobuf232\java>mvn package -Dmaven.test.skip=true
[INFO] Scanning for projects...
 ------------------------------------------------------------------------
………………
…………
……
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:55 min
[INFO] Finished at: 2023-06-11T13:32:50+08:00
[INFO] ------------------------------------------------------------------------
[WARNING]
[WARNING] Plugin validation issues were detected in 9 plugin(s)
[WARNING]
[WARNING]  * org.apache.maven.plugins:maven-jar-plugin:2.6
[WARNING]  * org.apache.maven.plugins:maven-compiler-plugin:3.6.1
[WARNING]  * org.apache.maven.plugins:maven-resources-plugin:3.1.0
[WARNING]  * org.jetbrains.kotlin:kotlin-maven-plugin:1.6.0
[WARNING]  * org.codehaus.mojo:build-helper-maven-plugin:1.10
[WARNING]  * org.codehaus.mojo:animal-sniffer-maven-plugin:1.20
[WARNING]  * org.apache.maven.plugins:maven-antrun-plugin:3.0.0
[WARNING]  * org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5
[WARNING]  * org.apache.felix:maven-bundle-plugin:3.0.1
[WARNING]
[WARNING] For more or less details, use 'maven.plugin.validation' property with one of the values (case insensitive): [BRIEF, DEFAULT, VERBOSE]
[WARNING]
    之所以把中间的错误贴出来,其实只是分享一下我解决这个问题的方式,实际上在不同的环境、不同的各类软件版本中,可能出现多种多样的错误,一般而言,无非就是缺少了代码、缺少了dependency、plugin之类的错误,找出来,想办法补上或者绕过去就好了。 

(6)PS:安装 protobuf 2.5.0

    同样,在我们编译hadoop之前,需要按照官方要求,将protobuf 2.5.0给编译了,并且install到本地仓库中去。过程嘛,和上面23.2版本的编译是一样的。这里以直接下载protoc.exe 2.5.0版本的为例。

     同样,在protobuf的根目录、src子目录中放入protoc.exe(当然必须和protobuf源代码版本一致,2.5.0版的),然后设好路径,新启动cmd,查看如下:        
Microsoft Windows [版本 10.0.19044.1288]
(c) Microsoft Corporation。保留所有权利。

C:\Users\pig>protoc --version
libprotoc 2.5.0

C:\Users\pig>
    然后到protobuf的java目录 下,直接mvn install:
Microsoft Windows [版本 10.0.19044.1288]
(c) Microsoft Corporation。保留所有权利。

C:\Users\pig>cd c:\protobuf

c:\protobuf>cd java

c:\protobuf\java>mvn install -Dmaven.test.skip=true
     然后一把过:

** (7)其它方法**

     其实不止使用Visual Studio和maven可以编译protobuf。作为跨平台软件,protobuf还支持其它好多种编译方式,这些编译方式都在对应子目录的Readme.MD中详细记载。

    比如参考https://github.com/protocolbuffers/protobuf/blob/main/src/README.md ,可以使用bazel在centos7上进行编译:

    首先安装bazel:

** dnf install dnf-plugins-core**

[root@pighost abseil-cpp]# yum install dnf-plugins-core -y
上次元数据过期检查:2:09:55 前,执行于 2023年06月03日 星期六 23时34分20秒。
软件包 dnf-plugins-core-4.0.21-11.el8.noarch 已安装。
依赖关系解决。
============================================================================================================================================
 软件包                                       架构                       版本                              仓库                        大小
============================================================================================================================================
升级:
 dnf-plugins-core                             noarch                     4.0.21-18.el8                     baseos                      75 k
 python3-dnf-plugins-core                     noarch                     4.0.21-18.el8                     baseos                     258 k

事务概要
============================================================================================================================================
升级  2 软件包

** dnf copr enable vbatts/bazel**

    这一句实际我没有执行

** yum install bazel5**

[root@pighost abseil-cpp]# yum  install bazel5 -y
上次元数据过期检查:0:00:17 前,执行于 2023年06月04日 星期日 01时44分57秒。
依赖关系解决。
============================================================================================================================================
 软件包                        架构        版本                                      仓库                                              大小
============================================================================================================================================
安装:
 bazel5                        x86_64      5.4.0-0.el8                               copr:copr.fedorainfracloud.org:vbatts:bazel       28 M
安装依赖关系:
 copy-jdk-configs              noarch      4.0-2.el8                                 appstream                                         31 k
 java-11-openjdk               x86_64      1:11.0.18.0.9-0.3.ea.el8                  appstream                                        468 k
 java-11-openjdk-devel         x86_64      1:11.0.18.0.9-0.3.ea.el8                  appstream                                        3.4 M
 java-11-openjdk-headless      x86_64      1:11.0.18.0.9-0.3.ea.el8                  appstream                                         41 M
 javapackages-filesystem       noarch      5.3.0-1.module_el8.0.0+11+5b8c10bd        appstream                                         30 k
 lksctp-tools                  x86_64      1.0.18-3.el8                              baseos                                           100 k
 ttmkfdir                      x86_64      3.0.9-54.el8                              appstream                                         62 k
 tzdata-java                   noarch      2023c-1.el8                               appstream                                        267 k
 xorg-x11-fonts-Type1          noarch      7.5-19.el8                                appstream                                        522 k
启用模块流:
 javapackages-runtime                      201801             
    然后编译,在protobuf的根目录下执行:

** bazel build :protoc :protobuf**

    生成结果位于bazel-bin目录下。        

三、编译Hadoop

    完成protobuf 2.5.0的install后,就可以开始编译hadoop了。

1. 下载Hadoop源码

    直接到官方下载源码包,解压到需要的目录下面就行 ![](https://img-blog.csdnimg.cn/578715773c0c49a5bc2ed951000f7d21.png)

    我是在c盘下建立了一个hadoop目录,然后把源代码解压进去了。需要注意的是,最后那个hadoop-yarn-project的子目录很深,所以要尽可能把代码解压到靠近根目录的位置。否则,在我正版的win11家庭版上,会因为目录长度限制而无法解压(其实似乎无论如何都规避不了,不过好在我们最后并不需要这个子目录里面的东西,所以直接无视了);而在我下载的未激活版本的windows专业版上,这个问题不存在。      ![](https://img-blog.csdnimg.cn/a55be06e65864fae95a81e16d82c366b.png)

2. 直接编译

c:\hadoop>mvn package -Dmaven.test.skip=true
[INFO] Scanning for projects...

(1)没有安装VS2019导致的错误

    直接上手package,一阵子以后,收到以下错误:
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (convert-ms-winutils) on project hadoop-common: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :hadoop-common
    每次收到这个错误都是崩溃的。因为引起它的原因千奇百怪。这次的原因,是因为在虚拟机里面,我偷懒没有安装Visual Studio,导致无法编译winutils。

    所以下载并安装——本来是为了避免在虚拟机中如此痛苦——结果也没避开。

使用Visual Studio的命令行工具进去重新mvn pakcage,这次成功过了winutils这一关,并且顺利生成了WinUtils.exe。其实我们需要的仅此而已。

(2)没有安装git-bash导致的错误

** ** 不过接着往下,我们又遇到了新的错误:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (common-test-bats-driver) on project hadoop-common: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "bash" (in directory "C:\hadoop\hadoop-common-project\hadoop-common\src\test\scripts"): CreateProcess error=2, 系统找不到指定的文件。
[ERROR] around Ant part ...<exec failonerror="true" dir="src/test/scripts" executable="bash">... @ 4:69 in C:\hadoop\hadoop-common-project\hadoop-common\target\antrun\build-main.xml
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :hadoop-common
     这个错误出现的原因,是因为没有安装git-bash,看ERROR报出的信息,实际就是maven使用antrun插件试图去执行build-main.xml文档(实际使用bash调用对应的sh脚本)时,无法找到bash文件。

    所以,记得安装git-bash,并且正确的设置PATH,使编译器能够找到git/bin/bash。

(3)跳过不太重要的部分

    接着往下,马上又碰到新的错误:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec failonerror="true" dir="C:\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native" executable="cmake">... @ 5:123 in C:\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :hadoop-hdfs-native-client
   从错误调试日志来看,应该是使用cmake和msbuild生成项目文件时,都出现了项目文件不存在的情况(似乎这个版本在这个client处缺少源文件)
Execute:Java13CommandLauncher: Executing 'cmake' with arguments:
'C:\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client/src/'
'-DGENERATED_JAVAH=C:\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native/javah'
'-DJVM_ARCH_DATA_MODEL=64'
'-DHADOOP_BUILD=1'
'-DREQUIRE_FUSE=false'
'-DREQUIRE_VALGRIND=false'
'-A'
'x64'

The ' characters around the executable and arguments are
not part of the command.
     [exec] CMake Warning (dev) in CMakeLists.txt:
     [exec]   No project() command is present.  The top-level CMakeLists.txt file must
     [exec]   contain a li-- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.19044.
     [exec] teral, direct call to the project() command.  Add a line of
     [exec]   code such as
     [exec]
     [exec]     project(ProjectName)
     [exec]
     [exec]   near the top of the file, but after cmake_minimum_required().
     [exec]
     [exec]   CMake is pretending there is a "project(Project)" command on the first
     [exec]   line.
     [exec] This warning is for project developers.  Use -Wno-dev to suppress it.
     [exec]
     [exec] CMake Warning (dev) in CMakeLists.txt:
     [exec]   cmake_minimum_required() should be called prior to this top-level project()
     [exec]   call.  Please see the cmake-commands(7) manual for usage documentation of
     [exec]   both commands.
     [exec] This warning is for project developers.  Use -Wno-dev to suppress it.
     [exec]
     [exec] CUSTOM_OPENSSL_PREFIX =
     [exec] -- Performing Test THREAD_LOCAL_SUPPORTED
     [exec] Cannot find a usable OpenSSL library. OPENSSL_LIBRARY=OPENSSL_LIBRARY-NOTFOUND, OPENSSL_INCLUDE_DIR=OPENSSL_INCLUDE_DIR-NOTFOUND, CUSTOM_OPENSSL_LIB=, CUSTOM_OPENSSL_PREFIX=, CUSTOM_OPENSSL_INCLUDE=
     [exec] CMake Deprecation Warning at main/native/libhdfspp/CMakeLists.txt:29 (cmake_minimum_required):
     [exec]   Compatibility with CMake < 2.8.12 will be removed from a future version of
     [exec]   CMake.
     [exec]
     [exec]   Update the VERSION argument <min> value or use a ...<max> suffix to tell-- Performing Test THREAD_LOCAL_SUPPORTED - Success
     [exec]
     [exec]   CMake that the project does not need compatibility with older versions.
     [exec]
     [exec]
     [exec] -- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
     [exec] CMake Error at C:/CMake/share/cmake-3.26/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
     [exec]   Could NOT find OpenSSL, tr-- Configuring incomplete, errors occurred!
     [exec] y to set the path to OpenSSL root folder in the
     [exec]   system variable OPENSSL_ROOT_DIR (missing: OPENSSL_CRYPTO_LIBRARY
     [exec]   OPENSSL_INCLUDE_DIR)
     [exec] Call Stack (most recent call first):
     [exec]   C:/CMake/share/cmake-3.26/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
     [exec]   C:/CMake/share/cmake-3.26/Modules/FindOpenSSL.cmake:670 (find_package_handle_standard_args)
     [exec]   main/native/libhdfspp/CMakeLists.txt:44 (find_package)
     [exec]
     [exec]
     [exec] Result: 1
     [exec] Current OS is Windows 10
     [exec] Executing 'msbuild' with arguments:
     [exec] 'ALL_BUILD.vcxproj'
     [exec] '/nologo'
     [exec] '/p:Configuration=RelWithDebInfo'
     [exec] '/p:LinkIncremental=false'
     [exec]
     [exec] The ' characters around the executable and arguments are
     [exec] not part of the command.
Execute:Java13CommandLauncher: Executing 'msbuild' with arguments:
'ALL_BUILD.vcxproj'
'/nologo'
'/p:Configuration=RelWithDebInfo'
'/p:LinkIncremental=false'

The ' characters around the executable and arguments are
not part of the command.
     [exec] MSBUILD : error MSB1009: 项目文件不存在。
    而且这个native-client底下的缺失较多,需要采取多种方法来绕过。具体可以参考

win10编译hadoop3.2.1_windows编译hadoop_冰上浮云的博客-CSDN博客。当然,最简单的绕过方法,是直接从下一个包开始编译:

    mvn package -Dmaven.test.skip=true -rf :hadoop-hdfs-nfs

    只要这个native-client不影响咱们啥就好;具体啥原因这里就不深究了。

    不过不知道是不是我们skip了test的原因……因为在上一次成功编译时,我使用的mvn test,没有遇到过类似问题。

(4)编译到中途,缺少denpendcy,或者pom路径不对

    先多说一句,如果遇上duplicate jar库,需要在mvn命令中添加clean参数。当然,这有可能把(3)中我们手工构建的目录删掉,没关系,碰到(3)错误的时候重新构建并去掉clean参数继续执行就行了。

    最终是需要mvn install package一起上的。

    然后在即将大功告成的某步,居然被告知ensure-jars-have-correct-contents.sh找不到文件:
[DEBUG] env: __VSCMD_PREINIT_PATH=C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\CMake\bin;C:\maven\bin;C:\protobuf;C:\Java\bin;C:\Jre\bin;C:\ant\bin;C:\Program Files\Git\cmd;C:\Program Files\Git\bin;C:\hadoop335\bin;C:\spark340\bin;C:\Users\pig\AppData\Local\Microsoft\WindowsApps;
[DEBUG] env: __VSCMD_SCRIPT_ERR_COUNT=0
[DEBUG] Executing command line: [bash, ensure-jars-have-correct-contents.sh, c:\m2repo\org\apache\hadoop\hadoop-client-api\3.3.5\hadoop-client-api-3.3.5.jar;c:\m2repo\org\apache\hadoop\hadoop-client-runtime\3.3.5\hadoop-client-runtime-3.3.5.jar]
/usr/bin/bash: ensure-jars-have-correct-contents.sh: No such file or directory
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Client Packaging Invariants 3.3.5:
[INFO]
[INFO] Apache Hadoop Client Packaging Invariants .......... FAILURE [  2.375 s]
[INFO] Apache Hadoop Client Test Minicluster .............. SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
[INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] Apache Hadoop Client Modules ....................... SKIPPED
[INFO] Apache Hadoop Tencent COS Support .................. SKIPPED
[INFO] Apache Hadoop Cloud Storage ........................ SKIPPED
[INFO] Apache Hadoop Cloud Storage Project ................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.406 s
[INFO] Finished at: 2023-06-11T22:10:37+08:00
[INFO] ------------------------------------------------------------------------
     根据调试信息顺藤摸瓜,直到是本地库C:\m2repo\org\apache\hadoop\hadoop-client-api\3.3.5目录下依赖库c:\m2repo\org\apache\hadoop\hadoop-client-runtime\3.3.5\hadoop-client-runtime-3.3.5.jar少了:

     一般而言,这里的jar文件都是可以在pom文件中指定,并由maven下载的。

    由于我们装到中途,改pom文件就比较麻烦,不如直接下jar包放到库里了。

     另外,这里bash用来执行的路径使用了相对路径ensure-jars-have-correct-contents.sh,但并没有正确指明相对的位置(比如使用环境变量);所以在hadoop\hadoop-client-modules\hadoop-client-check-invariants目录下找到pom文件,将其改为绝对地址:
        <executions>
          <execution>
            <id>check-jar-contents</id>
            <phase>integration-test</phase>
            <goals>
              <goal>exec</goal>
            </goals>
            <configuration>
              <executable>${shell-executable}</executable>
              <workingDirectory>${project.build.testOutputDirectory}</workingDirectory>
              <requiresOnline>false</requiresOnline>
              <arguments>
                <argument>C:\hadoop\hadoop-client-modules\hadoop-client-check-invariants\src\test\resources\ensure-jars-have-correct-contents.sh</argument>
                <argument>${hadoop-client-artifacts}</argument>
              </arguments>
            </configuration>
          </execution>
        </executions>

(5)Shade打包时jar包冲突

[INFO] Apache Hadoop Client Test Minicluster .............. FAILURE [02:13 min]
[INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
[INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] Apache Hadoop Client Modules ....................... SKIPPED
[INFO] Apache Hadoop Tencent COS Support .................. SKIPPED
[INFO] Apache Hadoop Cloud Storage ........................ SKIPPED
[INFO] Apache Hadoop Cloud Storage Project ................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:15 min
[INFO] Finished at: 2023-06-11T22:43:57+08:00
[INFO] ------------------------------------------------------------------------
[WARNING]
[WARNING] Plugin validation issues were detected in 9 plugin(s)
[WARNING]
[WARNING]  * org.apache.maven.plugins:maven-jar-plugin:2.5
[WARNING]  * org.apache.maven.plugins:maven-resources-plugin:2.6
[WARNING]  * org.apache.maven.plugins:maven-site-plugin:3.9.1
[WARNING]  * org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M1
[WARNING]  * org.apache.maven.plugins:maven-compiler-plugin:3.1
[WARNING]  * org.apache.maven.plugins:maven-shade-plugin:3.2.1
[WARNING]  * org.apache.maven.plugins:maven-antrun-plugin:1.7
[WARNING]  * org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1
[WARNING]  * com.google.code.maven-replacer-plugin:replacer:1.5.3
[WARNING]
[WARNING] For more or less details, use 'maven.plugin.validation' property with one of the values (case insensitive): [BRIEF, DEFAULT, VERBOSE]
[WARNING]
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project hadoop-client-minicluster: Error creating shaded jar: duplicate entry: META-INF/services/org.apache.hadoop.shaded.javax.websocket.ContainerProvider -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project hadoop-client-minicluster: Error creating shaded jar: duplicate entry: META-INF/services/org.apache.hadoop.shaded.javax.websocket.ContainerProvider
    这个一般就是打包时发现同一个class类在多个jar包里面了

    直接就地clean接着来
c:\hadoop>mvn clean install package -Dmaven.test.skip=true -rf :hadoop-client-minicluster

** (6)编译完成**

[INFO] Reactor Summary for Apache Hadoop HDFS-NFS 3.3.5:
[INFO]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 11.250 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 10.765 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.079 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.046 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 19.828 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 23.548 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.046 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 11.844 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 19.625 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  1.156 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [  3.375 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [  2.282 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 24.031 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  1.110 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [  3.109 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  0.672 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  0.797 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [  0.062 s]
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [  6.782 s]
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [  8.499 s]
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [  0.047 s]
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2  SUCCESS [  5.688 s]
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [  7.468 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [  1.219 s]
[INFO] Apache Hadoop YARN TimelineService DocumentStore ... SUCCESS [  8.031 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.032 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  0.875 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  0.250 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.062 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 13.141 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [  4.375 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  0.703 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [  4.562 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  2.235 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  2.531 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.516 s]
[INFO] Apache Hadoop YARN Services ........................ SUCCESS [  0.031 s]
[INFO] Apache Hadoop YARN Services Core ................... SUCCESS [  5.344 s]
[INFO] Apache Hadoop YARN Services API .................... SUCCESS [  0.968 s]
[INFO] Apache Hadoop YARN Application Catalog ............. SUCCESS [  0.032 s]
[INFO] Apache Hadoop YARN Application Catalog Webapp ...... SUCCESS [01:52 min]
[INFO] Apache Hadoop YARN Application Catalog Docker Image  SUCCESS [  0.046 s]
[INFO] Apache Hadoop YARN Application MaWo ................ SUCCESS [  0.063 s]
[INFO] Apache Hadoop YARN Application MaWo Core ........... SUCCESS [  1.000 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.047 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  0.078 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [  0.031 s]
[INFO] Apache Hadoop YARN CSI ............................. SUCCESS [ 10.375 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [  0.094 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  0.203 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [  1.266 s]
[INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [  0.281 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  1.859 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  0.094 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  2.266 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  2.031 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [  0.062 s]
[INFO] Apache Hadoop Dynamometer Workload Simulator ....... SUCCESS [  0.657 s]
[INFO] Apache Hadoop Dynamometer Cluster Simulator ........ SUCCESS [  1.234 s]
[INFO] Apache Hadoop Dynamometer Block Listing Generator .. SUCCESS [  0.375 s]
[INFO] Apache Hadoop Dynamometer Dist ..................... SUCCESS [  0.141 s]
[INFO] Apache Hadoop Dynamometer .......................... SUCCESS [  0.031 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  0.328 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  0.469 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  2.391 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  1.687 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  0.437 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  0.422 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  0.032 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 35.218 s]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [  1.938 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [  7.593 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [  2.688 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  1.813 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [  1.640 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  1.641 s]
[INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [  0.656 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  0.141 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  0.062 s]
[INFO] Apache Hadoop Common Benchmark ..................... SUCCESS [ 10.490 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.031 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [ 59.321 s]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [ 50.175 s]
[INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [  0.798 s]
[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [01:12 min]
[INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [  0.110 s]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [  0.515 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [  0.063 s]
[INFO] Apache Hadoop Client Modules ....................... SUCCESS [  0.031 s]
[INFO] Apache Hadoop Tencent COS Support .................. SUCCESS [  2.281 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  1.313 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [  0.031 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  10:00 min
[INFO] Finished at: 2023-06-11T20:21:34+08:00
[INFO] ------------------------------------------------------------------------
[WARNING]
[WARNING] Plugin validation issues were detected in 22 plugin(s)
[WARNING]
[WARNING]  * org.apache.maven.plugins:maven-assembly-plugin:2.4
[WARNING]  * org.apache.maven.plugins:maven-resources-plugin:2.6
[WARNING]  * org.codehaus.mojo:findbugs-maven-plugin:3.0.5
[WARNING]  * com.github.eirslett:frontend-maven-plugin:1.11.2
[WARNING]  * org.apache.maven.plugins:maven-compiler-plugin:3.1
[WARNING]  * org.apache.maven.plugins:maven-war-plugin:3.1.0
[WARNING]  * org.apache.maven.plugins:maven-shade-plugin:3.2.1
[WARNING]  * com.github.klieber:phantomjs-maven-plugin:0.7
[WARNING]  * com.google.code.maven-replacer-plugin:replacer:1.5.3
[WARNING]  * org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1
[WARNING]  * org.apache.maven.plugins:maven-dependency-plugin:3.0.2
[WARNING]  * org.apache.maven.plugins:maven-source-plugin:2.3
[WARNING]  * org.codehaus.mojo:build-helper-maven-plugin:1.9
[WARNING]  * org.apache.maven.plugins:maven-jar-plugin:2.5
[WARNING]  * org.apache.avro:avro-maven-plugin:1.7.7
[WARNING]  * org.codehaus.mojo:license-maven-plugin:1.10
[WARNING]  * org.apache.maven.plugins:maven-site-plugin:3.9.1
[WARNING]  * org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M1
[WARNING]  * org.xolstice.maven.plugins:protobuf-maven-plugin:0.5.1
[WARNING]  * com.github.searls:jasmine-maven-plugin:2.1
[WARNING]  * org.apache.maven.plugins:maven-antrun-plugin:1.7
[WARNING]  * org.apache.hadoop:hadoop-maven-plugins:3.3.5
[WARNING]
[WARNING] For more or less details, use 'maven.plugin.validation' property with one of the values (case insensitive): [BRIEF, DEFAULT, VERBOSE]
[WARNING]
    如果编译成功,长这样。

** (7)还不算完**

    虽然编译成功了,我们还是只能得到一大堆的jar,也就是说,如果需求是改造hadoop代码,替换原来的jar包,那么到这一步可以结束了。但是如果需求是得到我们自己的hadoop安装包,还需要本地重新编译一下:

    使用本地编译命令:
mvn package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
或者
mvn package -Pdist,native-win -DskipTests -Dtar -Dmaven.javadoc.skip=true
    从Maven Embedder - Maven CLI Options Reference (apache.org)  语焉不详的说明来看,我认为是-Pdist 和-Dtar起的作用。也就是说生成分发包和tar包(估计是根据不同的编译平台生成不同的结果)……不深究了,总之管用就行。

    这一轮编译完成后,才算是能看到可部署的Hadoop文件,在C:\hadoop\hadoop-dist\target\hadoop-3.3.5下:

    弄完以后发现一篇写得比较详尽的文章,参考价值较高:Compile and Build Hadoop 3.2.1 on Windows 10 Guide (kontext.tech)

四、Hadoop + Spark

    当然,如果你不太相信自己编译的这个Hadoop,完全可以官网上下一个已经编译好的,至少Spark我们就打算这么干了。

    把从官网下载的binary版本的压缩包解压到主机上:        ![](https://img-blog.csdnimg.cn/da1fd3c538934b7fb57300e6d776fd01.png)

(1)设置环境变量

    设置好环境变量。对于hadoop和spark来说,需要JAVA_HOME、HADOOP_HOME、SPARK_HOME3个环境变量,还需要配置PATH环境变量以便调用hdfs、spark-shell之类的命令。

     设置前改个名字,避免出现空格横线之类特别的字符。

     在PATH变量中,主要加入HADOOP和SPARK的bin目录位置,因为我们要调用的命令都在这里,sbin里面主要是用于管理集群的工具,所以无所谓。

(2)运行看看

      设置好后,直接运行该spark-shell可以发现,一个是前面会提示缺失winutils.exe,后面也不能“很干净”的退出。

    当然,如果只做交互式分析,而且没有强迫症的话,这也没啥。

    但是,如果是编程, 如下的这一句执行下来,你会发现目录下仅仅多了一个空的result目录,里面没有任何结果。这就是因为本应由winutils提供的接口不存在所导致的。

(3)拷贝winutils

    解决办法也很简单,直接把我们刚才编译产生的winutils拷贝到hadoop的bin目录下即可。生成的文件在\hadoop\hadoop-common-project\hadoop-common\target\bin下面:

然后重新开启一个cmd执行spark-shell:

     可以看到winutils的问题已经解决了。

     统计结果也顺利生成。

五、搭建hadoop + Spark + VSCODE

    写了一整天,实在写不动了。VSCODE的搭建没有什么幺蛾子了,和前面几篇讲的方法一样,不赘述。
标签: centos hadoop linux

本文转载自: https://blog.csdn.net/lhyzws/article/details/131144731
版权归原作者 lhyzws 所有, 如有侵权,请联系我们删除。

“CENTOS上的网络安全工具(二十四)Windows下的Hadoop+Spark编程环境构建”的评论:

还没有评论