0
0

Hadoop学习笔记之HDFS

Hadoop学习笔记之HDFS

HDFS （Hadoop Distributed File System）

分布式存储系统

支持海量数据的存储，成百上千的计算机组成存储集群，HDFS可以运行在低成本的硬件之上，具有的高容错、高可靠性、高可扩展性、高吞吐率等特征，非常适合大规模数据集上的应用。

优点

  • 高容错性
  • 适合批处理
  • 适合大数据处理
  • 流式文件访问
  • 可构建在廉价机器上

缺点

  • 不适合低延迟数据访问
  • 不适合小文件存取
  • 不适合并发写入、文件随机修改

HDFS操作

命令操作HDFS

# 显示目录 / 下的文件
hdfs dfs -ls /
# 新建文件夹，绝对路径
hdfs dfs -mkdir /test
# 上传文件
hdfs dfs -put test.txt /test/
# 下载文件
hdfs dfs -get /test/test.txt
# 输出文件内容
hdfs dfs -cat /test/test.txt

Web端操作HDFS

打开http://192.168.9.200:9870/，可以对HDFS进行操作

image-20220807170359131

文件在这里管理

image-20220807170408993

可视化操作还是简单一些

image-20220807170833704

Java操作HDFS

  1. 添加依赖
<dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>3.3.2</version></dependency>
  1. 建立连接
publicvoidsetUp()throwsException{System.out.println("开始建立与HDFS的连接");
    configuration =newConfiguration();
    fileSystem =FileSystem.get(newURI(HDFS_PATH), configuration,"hadoop");}
  1. 文件操作
Configuration configuration =null;FileSystem fileSystem =null;publicstaticfinalStringHDFS_PATH="hdfs://192.168.9.200:9000";/**
     * 在 hdfs中新建文件夹
     *
     * @throws Exception
     */@Testpublicvoidmkdir()throwsException{
        fileSystem.mkdirs(newPath("/JavaDemo/test"));}/**
     * 创建文件
     *
     * @throws Exception
     */@Testpublicvoidcreate()throwsException{FSDataOutputStream outputStream = fileSystem.create(newPath("/JavaDemo/test/haha.txt"));
        outputStream.write("hello bigdata from javaDemo".getBytes());
        outputStream.flush();
        outputStream.close();}/**
     * 查看文件 hdfs -fs -cat file
     *
     * @throws Exception
     */@Testpublicvoidcat()throwsException{FSDataInputStream in = fileSystem.open(newPath("/JavaDemo/test/haha.txt"));IOUtils.copyBytes(in,System.out,1024);
        in.close();}/**
     * 重命名文件
     *
     * @throws Exception
     */@Testpublicvoidrename()throwsException{Path oldPath =newPath("/JavaDemo/test/haha.txt");Path newPath =newPath("/JavaDemo/test/hehe.txt");
        fileSystem.rename(oldPath, newPath);}/**
     * 上传文件到HDFS
     *
     * @throws Exception
     */@TestpublicvoidcopyFromLocalFile()throwsException{Path loacalPath =newPath("hello.txt");Path hdfsPath =newPath("/");
        fileSystem.copyFromLocalFile(loacalPath, hdfsPath);}/**
     * 上传文件到HDFS带进度信息
     *
     * @throws Exception
     */@TestpublicvoidcopyFromLocalFileWithProgress()throwsException{InputStream in =newBufferedInputStream(Files.newInputStream(newFile("hbase-2.2.7-bin.tar.gz").toPath()));FSDataOutputStream ouput = fileSystem.create(newPath("/JavaDemo/test/hbase-2.2.7-bin.tar.gz"),()->{System.out.print(".");});IOUtils.copyBytes(in, ouput,4096);}/**
     * 下载文件到HDFS
     *
     * @throws Exception
     */@TestpublicvoidcopyToLocalFile()throwsException{Path hdfsPath =newPath("/JavaDemo/test/haha.txt");Path loacalPath =newPath("./haha.txt");//  useRawLocalFileSystem
        fileSystem.copyToLocalFile(false, hdfsPath, loacalPath,true);}/**
     * 查看某个目录下所有文件
     *
     * @throws Exception
     */@TestpublicvoidlistFiles()throwsException{FileStatus[] fileStatuses = fileSystem.listStatus(newPath("/"));for(FileStatus f : fileStatuses){String isDir = f.isDirectory()?"文件夹":"文件";short replication = f.getReplication();long len = f.getLen();String path = f.getPath().toString();System.out.println(isDir +"\t"+ replication +"\t"+ len +"\t"+ path);}}/**
     * 删除文件
     *
     * @throws Exception
     */@Testpublicvoiddelete()throwsIOException{
        fileSystem.delete(newPath("/JavaDemo/haha.txt"),true);}
  1. 关闭连接
publicvoidtearDown(){
    configuration =null;
    fileSystem =null;System.out.println("关闭与HDFS的连接");}

完整测试文件

importorg.apache.hadoop.conf.Configuration;importorg.apache.hadoop.fs.*;importorg.apache.hadoop.io.IOUtils;importorg.junit.After;importorg.junit.Before;importorg.junit.Test;importjava.io.*;importjava.net.URI;importjava.nio.file.Files;/**
 * @author Gettler
 * Java 操作HDFS
 */publicclassHdfsDemo{Configuration configuration =null;FileSystem fileSystem =null;publicstaticfinalStringHDFS_PATH="hdfs://192.168.9.200:9000";/**
     * 在 hdfs中新建文件夹
     *
     * @throws Exception
     */@Testpublicvoidmkdir()throwsException{
        fileSystem.mkdirs(newPath("/JavaDemo/test"));}/**
     * 创建文件
     *
     * @throws Exception
     */@Testpublicvoidcreate()throwsException{FSDataOutputStream outputStream = fileSystem.create(newPath("/JavaDemo/test/haha.txt"));
        outputStream.write("hello bigdata from javaDemo".getBytes());
        outputStream.flush();
        outputStream.close();}/**
     * 查看文件 hdfs -fs -cat file
     *
     * @throws Exception
     */@Testpublicvoidcat()throwsException{FSDataInputStream in = fileSystem.open(newPath("/JavaDemo/test/haha.txt"));IOUtils.copyBytes(in,System.out,1024);
        in.close();}/**
     * 重命名文件
     *
     * @throws Exception
     */@Testpublicvoidrename()throwsException{Path oldPath =newPath("/JavaDemo/test/haha.txt");Path newPath =newPath("/JavaDemo/test/hehe.txt");
        fileSystem.rename(oldPath, newPath);}/**
     * 上传文件到HDFS
     *
     * @throws Exception
     */@TestpublicvoidcopyFromLocalFile()throwsException{Path loacalPath =newPath("hello.txt");Path hdfsPath =newPath("/");
        fileSystem.copyFromLocalFile(loacalPath, hdfsPath);}/**
     * 上传文件到HDFS带进度信息
     *
     * @throws Exception
     */@TestpublicvoidcopyFromLocalFileWithProgress()throwsException{InputStream in =newBufferedInputStream(Files.newInputStream(newFile("hbase-2.2.7-bin.tar.gz").toPath()));FSDataOutputStream ouput = fileSystem.create(newPath("/JavaDemo/test/hbase-2.2.7-bin.tar.gz"),()->{System.out.print(".");});IOUtils.copyBytes(in, ouput,4096);}/**
     * 下载文件到HDFS
     *
     * @throws Exception
     */@TestpublicvoidcopyToLocalFile()throwsException{Path hdfsPath =newPath("/JavaDemo/test/haha.txt");Path loacalPath =newPath("./haha.txt");//  useRawLocalFileSystem
        fileSystem.copyToLocalFile(false, hdfsPath, loacalPath,true);}/**
     * 查看某个目录下所有文件
     *
     * @throws Exception
     */@TestpublicvoidlistFiles()throwsException{FileStatus[] fileStatuses = fileSystem.listStatus(newPath("/"));for(FileStatus f : fileStatuses){String isDir = f.isDirectory()?"文件夹":"文件";short replication = f.getReplication();long len = f.getLen();String path = f.getPath().toString();System.out.println(isDir +"\t"+ replication +"\t"+ len +"\t"+ path);}}/**
     * 删除文件
     *
     * @throws Exception
     */@Testpublicvoiddelete()throwsIOException{
        fileSystem.delete(newPath("/JavaDemo/haha.txt"),true);}//测试之前执行的代码@BeforepublicvoidsetUp()throwsException{System.out.println("开始建立与HDFS的连接");
        configuration =newConfiguration();
        fileSystem =FileSystem.get(newURI(HDFS_PATH), configuration,"hadoop");}//测试之完执行的代码@AfterpublicvoidtearDown(){
        configuration =null;
        fileSystem =null;System.out.println("关闭与HDFS的连接");}}

9999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999ssssssssss999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999ssssssssss999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999ssssssssss999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999ssssssssss999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999ssssssssss99999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999

标签： hadoop hdfs 学习
本文转载自: https://blog.csdn.net/qq_46039856/article/details/126214976
版权归原作者 Gettler•Main 所有， 如有侵权，请联系我们删除。
登录后发布评论

“Hadoop学习笔记之HDFS”的评论:

还没有评论
关于作者
...
overfit同步小助手
文章同步
相关阅读

网络安全法-网络运行安全

使用selenium/drissionpage时如何阻止chrome自动跳转http到https

docker desktop 里部署的Open WebUI 管理员密码忘记了的处理方法

在ubuntu20.04中搭建onsite比赛运行环境

利用开源的低代码表单设计器FcDesigner高效管理和渲染复杂表单结构

Kafka学习笔记

【前端】浏览器输入url到页面呈现发生了什么？

文章导航