0


13、HDFS Snapshot快照

Hadoop系列文章目录

1、hadoop3.1.4简单介绍及部署、简单验证
2、HDFS操作 - shell客户端
3、HDFS的使用(读写、上传、下载、遍历、查找文件、整个目录拷贝、只拷贝文件、列出文件夹下文件、删除文件及目录、获取文件及文件夹属性等)-java
4、HDFS-java操作类HDFSUtil及junit测试(HDFS的常见操作以及HA环境的配置)
5、HDFS API的RESTful风格–WebHDFS
6、HDFS的HttpFS-代理服务
7、大数据中常见的文件存储格式以及hadoop中支持的压缩算法
8、HDFS内存存储策略支持和“冷热温”存储
9、hadoop高可用HA集群部署及三种方式验证
10、HDFS小文件解决方案–Archive
11、hadoop环境下的Sequence File的读写与合并
12、HDFS Trash垃圾桶回收介绍与示例
13、HDFS Snapshot快照
14、HDFS 透明加密KMS
15、MapReduce介绍及wordcount
16、MapReduce的基本用法示例-自定义序列化、排序、分区、分组和topN
17、MapReduce的分区Partition介绍
18、MapReduce的计数器与通过MapReduce读取/写入数据库示例
19、Join操作map side join 和 reduce side join
20、MapReduce 工作流介绍
21、MapReduce读写SequenceFile、MapFile、ORCFile和ParquetFile文件
22、MapReduce使用Gzip压缩、Snappy压缩和Lzo压缩算法写文件和读取相应的文件
23、hadoop集群中yarn运行mapreduce的内存、CPU分配调度计算与优化


文章目录


一、介绍

1、快照介绍

快照(Snapshot)是数据存储的某一时刻的状态记录;与备份不同,备份(Backup)则是数据存储的某一个时刻的副本。
HDFS Snapshot快照是整个文件系统或某个目录在某个时刻的镜像。
该镜像并不会随着源目录的改变而进行动态的更新。

2、快照作用

  • 数据恢复 对重要目录进行创建snapshot的操作,当用户误操作时,可以通过snapshot来进行相关的恢复操作
  • 数据备份 使用snapshot来进行整个集群或者某些目录、文件的备份 管理员以某个时刻的snapshot作为备份的起始结点,然后通过比较不同备份之间差异性,来进行增量备份
  • 数据测试 可以临时的为用户针对要操作的数据来创建一个snapshot,然后让用户在对应的snapshot上进行相关的实验和测试

3、HDFS快照功能

  • HDFS快照不是数据的简单拷贝,只做差异的记录
  • 对于大多不变的数据,你所看到的数据其实是当前物理路径所指的内容,而发生变更的inode数据才会被快照额外拷贝,也就是所说的差异拷贝
  • inode指索引节点,用来存放文件及目录的基本信息,包含时间、名称、拥有者、所在组等
  • HDFS快照不会复制datanode中的块,只记录了块列表和文件大小
  • HDFS快照不会对常规HDFS操作产生不利影响,修改记录按逆时针顺序进行,因此可以直接访问当前数据。通过从当前数据中减去修改来计算快照数据

二、使用及验证

1、快照功能开启

HDFS中可以针对整个文件系统或者某个目录创建快照,但是创建快照的前提是相应的目录开启快照的功能。如果针对没有启动快照功能的目录创建快照则会报错。

#启用快照功能:
hdfs dfsadmin -allowSnapshot /testsnapshot

[alanchan@server1 ~]$ hadoop fs -ls /testsnapshot
[alanchan@server1 ~]$ hadoop fs -createSnapshot  /testsnapshot test_snapshot
createSnapshot: Directory is not a snapshottable directory: /testsnapshot
[alanchan@server1 ~]$ hdfs dfsadmin -allowSnapshot /testsnapshot
Allowing snapshot on /testsnapshot succeeded
[alanchan@server1 ~]$ hadoop fs -createSnapshot  /testsnapshot test_snapshot
Created snapshot /testsnapshot/.snapshot/test_snapshot

2、快照功能禁用

HDFS中可以针对已经开启快照功能的目录进行禁用。禁用的前提是该目录的所有快照已经被删除。

#禁用快照功能:
hdfs dfsadmin -disallowSnapshot  /testsnapshot

3、快照操作相关命令

createSnapshot 创建快照
deleteSnapshot 删除快照
renameSnapshot 重命名快照
lsSnapshottableDir 列出可以快照目录列表
snapshotDiff 获取快照差异报告

[alanchan@server1 ~]$ hadoop dfs
WARNING: Use of this script to execute dfs is deprecated.
WARNING: Attempting to execute replacement "hdfs dfs" instead.

Usage: hadoop fs [generic options]...
        [-createSnapshot <snapshotDir>[<snapshotName>]][-deleteSnapshot <snapshotDir><snapshotName>][-renameSnapshot <snapshotDir><oldName><newName>]...
command[genericOptions][commandOptions][alanchan@server1 ~]# hdfs lsSnapshottableDir[alanchan@server1 ~]# hdfs snapshotDiff <path> <fromSnapshot> <toSnapshot>

4、HDFS Snapshot使用

1)、开启指定目录快照功能

[alanchan@server1 ~]$ hdfs dfsadmin -allowSnapshot /testsnapshot
Allowing snapshot on /testsnapshot succeeded

2)、创建快照

#系统自动生成快照名称
hdfs dfs -createSnapshot /testsnapshot
#指定名称创建快照
hdfs dfs -createSnapshot /testsnapshot test_snapshot

[alanchan@server1 ~]$ hadoop fs -createSnapshot  /testsnapshot test_snapshot
Created snapshot /testsnapshot/.snapshot/test_snapshot

在这里插入图片描述

3)、Web页面浏览快照

在这里插入图片描述

4)、重命名快照

#重命名快照
hdfs dfs -renameSnapshot /testsnapshot test_snapshot rename_test_snapshot 

[alanchan@server1 ~]$ hdfs dfs -renameSnapshot /testsnapshot test_snapshot rename_test_snapshot 
[alanchan@server1 ~]$ hadoop fs -ls  /testsnapshot/.snapshot
Found 1 items
drwxr-xr-x   - alanchan supergroup          02022-09-13 09:38 /testsnapshot/.snapshot/rename_test_snapshot

5)、列出HDFS集群所有开启快照功能的目录

#列出HDFS集群所有开启快照功能的目录
hdfs lsSnapshottableDir

[alanchan@server1 ~]$ hdfs lsSnapshottableDir
drwxr-xr-x 0 alanchan supergroup 02022-09-13 09:38 165536 /testsnapshot

6)、快照间差异比较

准备快照

hdfs dfs -createSnapshot /testsnapshot snapshot1

#1、在当前文件夹下创建一个快照snapshot1[alanchan@server1 ~]$ hdfs dfs -createSnapshot /testsnapshot snapshot1
Created snapshot /testsnapshot/.snapshot/snapshot1
#2、在当前文件夹下上传一个文件,并再创建一个快照snapshot2[alanchan@server1 ~]$ cd /usr/local/bigdata/hadoop-3.1.4
[alanchan@server1 hadoop-3.1.4]$ echo12345>>1.txt
[alanchan@server1 hadoop-3.1.4]$ hadoop fs -put1.txt /testsnapshot
[alanchan@server1 hadoop-3.1.4]$ hadoop fs -ls /testsnapshot
Found 1 items
-rw-r--r--   3 alanchan supergroup          62022-09-13 10:05 /testsnapshot/1.txt
[alanchan@server1 hadoop-3.1.4]$ hdfs dfs -createSnapshot /testsnapshot snapshot2
Created snapshot /testsnapshot/.snapshot/snapshot2
#3、修改文件内容后,再创建一个快照snapshot3[alanchan@server1 hadoop-3.1.4]$ echo54321>>2.txt
[alanchan@server1 hadoop-3.1.4]$ hadoop fs -appendToFile2.txt /testsnapshot/1.txt
[alanchan@server1 hadoop-3.1.4]$ hadoop fs -cat /testsnapshot/1.txt
1234554321[alanchan@server1 hadoop-3.1.4]$ hdfs dfs -createSnapshot /testsnapshot snapshot3
Created snapshot /testsnapshot/.snapshot/snapshot3
#4、上传一个文件,再创建一个快照snapshot4[alanchan@server1 hadoop-3.1.4]$ hadoop fs -put README.txt /testsnapshot
[alanchan@server1 hadoop-3.1.4]$ hdfs dfs -createSnapshot /testsnapshot snapshot4
Created snapshot /testsnapshot/.snapshot/snapshot4
#5、删除一个文件后,再创建一个快照snapshot5[alanchan@server1 hadoop-3.1.4]$ hadoop fs -rm /testsnapshot/README.txt
2022-09-13 10:09:31,863 INFO fs.TrashPolicyDefault: Moved: 'hdfs://HadoopHAcluster/testsnapshot/README.txt' to trash at: hdfs://HadoopHAcluster/user/alanchan/.Trash/Current/testsnapshot/README.txt
[alanchan@server1 hadoop-3.1.4]$ hdfs dfs -createSnapshot /testsnapshot snapshot5
Created snapshot /testsnapshot/.snapshot/snapshot5
[alanchan@server1 hadoop-3.1.4]$ 

在这里插入图片描述

7)、比较快照差异

+ The file/directory has been created.
-  The file/directory has been deleted.
M  The file/directory has been modified.
R  The file/directory has been renamed.
#比较指定目录两个版本快照之间的差异
hdfs snapshotDiff /testsnapshot snapshot1 snapshot2

[alanchan@server1 hadoop-3.1.4]$ hdfs snapshotDiff /testsnapshot snapshot1 snapshot2
Difference between snapshot snapshot1 and snapshot snapshot2 under directory /testsnapshot:
M       .
+       ./1.txt

[alanchan@server1 hadoop-3.1.4]$ hdfs snapshotDiff /testsnapshot snapshot1 snapshot3
Difference between snapshot snapshot1 and snapshot snapshot3 under directory /testsnapshot:
M       .
+       ./1.txt

[alanchan@server1 hadoop-3.1.4]$ hdfs snapshotDiff /testsnapshot snapshot1 snapshot4
Difference between snapshot snapshot1 and snapshot snapshot4 under directory /testsnapshot:
M       .
+       ./1.txt
+       ./README.txt

[alanchan@server1 hadoop-3.1.4]$ hdfs snapshotDiff /testsnapshot snapshot1 snapshot5
Difference between snapshot snapshot1 and snapshot snapshot5 under directory /testsnapshot:
M       .
+       ./1.txt

[alanchan@server1 hadoop-3.1.4]$ hdfs snapshotDiff /testsnapshot snapshot2 snapshot3
Difference between snapshot snapshot2 and snapshot snapshot3 under directory /testsnapshot:
M       ./1.txt

[alanchan@server1 hadoop-3.1.4]$ hdfs snapshotDiff /testsnapshot snapshot2 snapshot4
Difference between snapshot snapshot2 and snapshot snapshot4 under directory /testsnapshot:
M       .
+       ./README.txt
M       ./1.txt

[alanchan@server1 hadoop-3.1.4]$ hdfs snapshotDiff /testsnapshot snapshot2 snapshot5
Difference between snapshot snapshot2 and snapshot snapshot5 under directory /testsnapshot:
M       ./1.txt

[alanchan@server1 hadoop-3.1.4]$ hdfs snapshotDiff /testsnapshot snapshot3 snapshot4
Difference between snapshot snapshot3 and snapshot snapshot4 under directory /testsnapshot:
M       .
+       ./README.txt

[alanchan@server1 hadoop-3.1.4]$ hdfs snapshotDiff /testsnapshot snapshot3 snapshot5
Difference between snapshot snapshot3 and snapshot snapshot5 under directory /testsnapshot:

[alanchan@server1 hadoop-3.1.4]$ hdfs snapshotDiff /testsnapshot snapshot4 snapshot5
Difference between snapshot snapshot4 and snapshot snapshot5 under directory /testsnapshot:
M       .
-       ./README.txt

[alanchan@server1 hadoop-3.1.4]$ 

8)、删除快照

 hdfs dfs -deleteSnapshot /testsnapshot rename_test_snapshot

9)、删除开启快照功能的目录

hadoop fs -rm -r /allenwoon
拥有快照的目录不允许被删除,某种程度上也保护了文件安全

[alanchan@server1 hadoop-3.1.4]$ hadoop fs -rm-r /testsnapshot
rm: Failed to move to trash: hdfs://HadoopHAcluster/testsnapshot: The directory /testsnapshot cannot be deleted since /testsnapshot is snapshottable and already has snapshots
标签: hdfs hadoop 大数据

本文转载自: https://blog.csdn.net/chenwewi520feng/article/details/130362505
版权归原作者 一瓢一瓢的饮 alanchan 所有, 如有侵权,请联系我们删除。

“13、HDFS Snapshot快照”的评论:

还没有评论