如何利用IDEA通过创建Maven项目来实现hadoop相关项目

前言

已经安装好IDEA，并且配置好JDK和maven

一、创建一个Maven项目

点击玩Finish之后，等待加载完项目（第一次加载需要等一段时间）

如图示，maven项目加载完毕，并将下图所示内容删除

二、导入hadoop相关依赖，即修改pom.xml文件

在</project>前的<build>节点下面增加如下内容：

<plugins>
   <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-jar-plugin</artifactId>
      <version>2.6</version>
      <configuration>
         <archive>
            <manifest>
               <!--main（）所在的类，注意修改-->
               <mainClass>join.joinMain</mainClass>
            </manifest>
         </archive>
      </configuration>
   </plugin>

注意：以下添加的依赖中<version>版本号</version>版本号，需要和虚拟机中hadoop的版本一致

添加hadoop-common依赖

<dependency>
   <groupId>org.apache.hadoop</groupId>
   <artifactId>hadoop-common</artifactId>
   <version>3.3.1</version>
</dependency>

添加hadoop-hdfs依赖

<dependency>
   <groupId>org.apache.hadoop</groupId>
   <artifactId>hadoop-hdfs</artifactId>
   <version>3.3.1</version>
   <scope>test</scope>
</dependency>

添加hadoop-mapreduce-client-core依赖

<dependency>
   <groupId>org.apache.hadoop</groupId>
   <artifactId>hadoop-mapreduce-client-core</artifactId>
   <version>3.3.1</version>
</dependency>

添加hadoop-client依赖

<dependency>
   <groupId>org.apache.hadoop</groupId>
   <artifactId>hadoop-client</artifactId>
   <version>3.3.1</version>
</dependency>

再导入完以上四个依赖之后，点击该图标进行刷新

点击右侧maven，我们可以看到如下内容：

三、编写代码

在src.main.java下面创建属于你自己的项目

四、打包，上传到集群上面运行

    由于我们本地Windows上面是没有hadoop环境的，所以我们编写完代码之后需要上传到集群上面。

1、打包

在pom.xml中添加如下依赖

<packaging>jar</packaging>

意思是，我们将此项目打包时，将其打成jar包。

2、点击package

等待一会出现如下，即表示打包成功

3、 将打包好的jar包上传到虚拟机中，建议直接放到hadoop的目录下面

4、运行jar包

hadoop jar sort-1.0-SNAPSHOT.jar sort.sortMain

命令详解：

sort-1.0-SNAPSHOT.jar：我们打jar包的名字

sortMain：我们所写项目的主类的路径

五、附带一个pom.xml文件

注意修改主类的路径，项目的名称

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
   <modelVersion>4.0.0</modelVersion>

   <groupId>org.example</groupId>
   <artifactId>sort</artifactId>
   <version>1.0-SNAPSHOT</version>
   <packaging>jar</packaging>

   <name>sort</name>
   <!-- FIXME change it to the project's website -->
   <url>http://www.example.com</url>

   <properties>
      <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
      <maven.compiler.source>1.7</maven.compiler.source>
      <maven.compiler.target>1.7</maven.compiler.target>
   </properties>

   <dependencies>
      <dependency>
         <groupId>junit</groupId>
         <artifactId>junit</artifactId>
         <version>4.11</version>
         <scope>test</scope>
      </dependency>

      <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
      <dependency>
         <groupId>org.apache.hadoop</groupId>
         <artifactId>hadoop-common</artifactId>
         <version>3.3.1</version>
      </dependency>


      <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-hdfs -->
      <dependency>
         <groupId>org.apache.hadoop</groupId>
         <artifactId>hadoop-hdfs</artifactId>
         <version>3.3.1</version>
         <scope>test</scope>
      </dependency>

      <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-mapreduce-client-core -->
      <dependency>
         <groupId>org.apache.hadoop</groupId>
         <artifactId>hadoop-mapreduce-client-core</artifactId>
         <version>3.3.1</version>
      </dependency>

      <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client -->
      <dependency>
         <groupId>org.apache.hadoop</groupId>
         <artifactId>hadoop-client</artifactId>
         <version>3.3.1</version>
      </dependency>

      <dependency>
         <groupId>log4j</groupId>
         <artifactId>log4j</artifactId>
         <version>1.2.17</version>
      </dependency>

   </dependencies>

   <build>
      <pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
         <plugins>
            <plugin>
               <groupId>org.apache.maven.plugins</groupId>
               <artifactId>maven-jar-plugin</artifactId>
               <version>2.6</version>
               <configuration>
                  <archive>
                     <manifest>
                        <!--main（）所在的类，注意修改-->
                        <mainClass>join.joinMain</mainClass>
                     </manifest>
                  </archive>
               </configuration>
            </plugin>
            <!-- clean lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#clean_Lifecycle -->
            <plugin>
               <artifactId>maven-clean-plugin</artifactId>
               <version>3.1.0</version>
            </plugin>
            <!-- default lifecycle, jar packaging: see https://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_jar_packaging -->
            <plugin>
               <artifactId>maven-resources-plugin</artifactId>
               <version>3.0.2</version>
            </plugin>
            <plugin>
               <artifactId>maven-compiler-plugin</artifactId>
               <version>3.8.0</version>
            </plugin>
            <plugin>
               <artifactId>maven-surefire-plugin</artifactId>
               <version>2.22.1</version>
            </plugin>
            <plugin>
               <artifactId>maven-jar-plugin</artifactId>
               <version>3.0.2</version>
            </plugin>
            <plugin>
               <artifactId>maven-install-plugin</artifactId>
               <version>2.5.2</version>
            </plugin>
            <plugin>
               <artifactId>maven-deploy-plugin</artifactId>
               <version>2.8.2</version>
            </plugin>
            <!-- site lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#site_Lifecycle -->
            <plugin>
               <artifactId>maven-site-plugin</artifactId>
               <version>3.7.1</version>
            </plugin>
            <plugin>
               <artifactId>maven-project-info-reports-plugin</artifactId>
               <version>3.0.0</version>
            </plugin>
         </plugins>
      </pluginManagement>
   </build>
</project>