文章目录
源码：
演示视频
1.1 项目背景
对于超大文件上传我们可能遇到以下问题
• 大文件直接上传，占用过多内存，可能导致内存溢出甚至系统崩溃
• 受网络环境影响，可能导致传输中断，只能重新传输
• 传输时间长，用户无法知道传输进度，用户体验不佳
1.2 项目目标
对于上述问题，我们需要对文件做分片传输。分片传输就是把文件分割成许多较小的文件，然后分多次上传，最后再完成合并。
受网络环境影响，我们还要实现断点续传，以节省传输时间和资源。断点续传就是已经上传或者下载过的文件分片不再传输。
对于已经上传过的文件，可以不再上传，实现秒传。秒传就是根据文件的唯一标识，确认是否需要上传。
实现多任务上传或下载。多任务就是同时多个文件上传或下载。
2.1 业务流程
用户上传文件的流程图如图1所示，用户首先选择要上传的文件，上传过程中可以选择暂停或继续上传。
用户上传文件的流程图如图2所示，用户首先可以浏览可以下载的文件列表，然后点击下载，下载过程中可以选择暂停或继续下载
2.2 系统用例
系统用例图如图3所示，用户可以上传文件，在文件上传过程中可以查看文件的上传进度和速度，也可以暂停或开始上传；用户可以查看已经上传过的，也就是可以下载的文件列表；用户可以下载文件，在下载过程中可以查看文件下载的速度和进度，用户可以暂停或开始下载。
2.3 系统总体功能
系统总体功能图如图4所示，分为上传和下载。上传包括秒传，分片上传，断点续传，多任务。下载包括分片下载，断点续传，多任务。
3.1 技术选型
后端：
• 语言：Java8
• 框架：SpringBoot2.6
• 开发工具：Idea 2021
前端：
• 语言：Html5、css3、JavaScript
• 框架：Vue3
• 开发工具：Vscode、Edge
数据库：
• mysql8
4.1 文件上传模块
文件上传模块的流程图如图6所示，顺序图如图7所示
首先前端读取文件生成文件的唯一标识MD5,这里采用常用的MD5生成框架：spark-md5.js。对于大文件一次性读取比较慢，而且容易造成浏览器崩溃，因此这里采用分片读取的方式计算MD5。
然后向服务器发送请求，查看该文件时候已经上传，如果已经上传，就提示用户已经秒传。
如果数据库中没有记录该文件，就表示该文件没有上传或没有上传完成，那么服务器就查询并返回记录的chunk分片列表。
async 和 await配可以实现等待异步函数计算完成
//计算文件的md5值functioncomputeMd5(file, uploadFile){returnnewPromise((resolve, reject)=>{//分片读取并计算md5const chunkTotal =100;//分片数const chunkSize = Math.ceil(file.size / chunkTotal);const fileReader =newFileReader();const md5 =newSparkMD5();let index =0;constloadFile=(uploadFile)=>{
uploadFile.parsePercentage.value =parseInt((index / file.size)*100);const slice = file.slice(index, index + chunkSize);
fileReader.readAsBinaryString(slice);};loadFile(uploadFile);
fileReader.onload=(e)=>{
md5.appendBinary(e.target.result);if(index < file.size){
index += chunkSize;loadFile(uploadFile);}else{// md5.end() 就是文件md5码resolve(md5.end());}};});}//检查文件是否存在functioncheckFile(md5){returnrequest({url:"/check",method:"get",params:{md5: md5,},});}//文件上传之前,el-upload自动触发asyncfunctionbeforeUpload(file){
console.log("2.上传文件之前");var uploadFile ={};
uploadFile.name = file.name;
uploadFile.size = file.size;
uploadFile.parsePercentage =ref(0);
uploadFile.uploadPercentage =ref(0);
uploadFile.uploadSpeed ="0 M/s";
uploadFile.chunkList =null;
uploadFile.file = file;
uploadFile.uploadingStop =false;
uploadFileList.value.push(uploadFile);var md5 =awaitcomputeMd5(file, uploadFile);//async 和 await配可以实现等待异步函数计算完成
uploadFile.md5 = md5;var res =awaitcheckFile(md5);//上传服务器检查，以确认是否秒传var data = res.data.data;if(!data.isUploaded){
uploadFile.chunkList = data.chunkList;
uploadFile.needUpload =true;}else{
uploadFile.needUpload =false;
uploadFile.uploadPercentage.value =100;
console.log("文件已秒传");ElMessage({showClose:true,message:"文件已秒传",type:"warning",});}}
前端分片请求文件，如果分片编号被包含在分片列表内，就标识该分片已经上传，跳过；反之，表示还未上传，那么前端通过file的slice方法分割文件，向服务端传递。同时在页面上显示上传进度和速度。
服务端，收到前端的分片文件后，通过Java的RandomAccess类（随机读写类），从文件的指定位置，写入指定字节，并记录chunk到数据库，如果是最后一个分片再记录file到数据库。
图6 文件上传流程图
图7 文件上传顺序图
前端代码
<template><divclass="main"><!-- 文件上传按钮 --><el-uploadaction="#":http-request="upload":before-upload="beforeUpload":show-file-list="false"><el-buttontype="primary">选择上传文件</el-button></el-upload><el-dividercontent-position="left">上传列表</el-divider><!-- 正在上传的文件列表 --><divclass="uploading"v-for="uploadFile in uploadFileList"><spanclass="fileName">{{ uploadFile.name }}</span><spanclass="fileSize">{{ formatSize(uploadFile.size) }}</span><divclass="parse"><span>解析进度： </span><el-progress:text-inside="true":stroke-width="16":percentage="uploadFile.parsePercentage"></el-progress></div><divclass="progress"><span>上传进度：</span><el-progress:text-inside="true":stroke-width="16":percentage="uploadFile.uploadPercentage"></el-progress><spanv-if="
(uploadFile.uploadPercentage > 0) &
(uploadFile.uploadPercentage < 100)
"><spanclass="uploadSpeed">{{ uploadFile.uploadSpeed }}</span><el-buttoncirclelink@click="changeUploadingStop(uploadFile)"><el-iconsize="20"v-if="uploadFile.uploadingStop == false"><VideoPause/></el-icon><el-iconsize="20"v-else><VideoPlay/></el-icon></el-button></span></div></div></div></template><scriptsetup>import emitter from"../utils/eventBus.js";import{ ElMessage }from"element-plus";import SparkMD5 from"spark-md5";import{ VideoPause, VideoPlay }from"@element-plus/icons-vue";import{ ref, reactive, getCurrentInstance, nextTick }from"vue";const{ appContext }=getCurrentInstance();const request = appContext.config.globalProperties.request;var uploadFileList =ref([]);//换算文件的大小单位functionformatSize(size){//size的单位大小kvar unit;var units =[" B"," K"," M"," G"];var pointLength =2;while((unit = units.shift())&& size >1024){
size = size /1024;}return((unit ==="B"? size
: size.toFixed(pointLength ===undefined?2: pointLength))+ unit
);}//计算文件的md5值functioncomputeMd5(file, uploadFile){returnnewPromise((resolve, reject)=>{//分片读取并计算md5const chunkTotal =100;//分片数const chunkSize = Math.ceil(file.size / chunkTotal);const fileReader =newFileReader();const md5 =newSparkMD5();let index =0;constloadFile=(uploadFile)=>{
uploadFile.parsePercentage.value =parseInt((index / file.size)*100);const slice = file.slice(index, index + chunkSize);
fileReader.readAsBinaryString(slice);};loadFile(uploadFile);
fileReader.onload=(e)=>{
md5.appendBinary(e.target.result);if(index < file.size){
index += chunkSize;loadFile(uploadFile);}else{// md5.end() 就是文件md5码resolve(md5.end());}};});}//检查文件是否存在functioncheckFile(md5){returnrequest({url:"/check",method:"get",params:{md5: md5,},});}//文件上传之前,el-upload自动触发asyncfunctionbeforeUpload(file){
console.log("2.上传文件之前");var uploadFile ={};
uploadFile.name = file.name;
uploadFile.size = file.size;
uploadFile.parsePercentage =ref(0);
uploadFile.uploadPercentage =ref(0);
uploadFile.uploadSpeed ="0 M/s";
uploadFile.chunkList =null;
uploadFile.file = file;
uploadFile.uploadingStop =false;
uploadFileList.value.push(uploadFile);var md5 =awaitcomputeMd5(file, uploadFile);//async 和 await配可以实现等待异步函数计算完成
uploadFile.md5 = md5;var res =awaitcheckFile(md5);//上传服务器检查，以确认是否秒传var data = res.data.data;if(!data.isUploaded){
uploadFile.chunkList = data.chunkList;
uploadFile.needUpload =true;}else{
uploadFile.needUpload =false;
uploadFile.uploadPercentage.value =100;
console.log("文件已秒传");ElMessage({showClose:true,message:"文件已秒传",type:"warning",});}}//点击暂停或开始上传functionchangeUploadingStop(uploadFile){
uploadFile.uploadingStop =!uploadFile.uploadingStop;if(!uploadFile.uploadingStop){uploadChunk(uploadFile.file,1, uploadFile);}}//上传文件,替换el-upload的actionfunctionupload(xhrData){var uploadFile =null;for(var i =0; i < uploadFileList.value.length; i++){if((xhrData.file.name == uploadFileList.value[i].name)&(xhrData.file.size == uploadFileList.value[i].size)){
uploadFile = uploadFileList.value[i];break;}}if(uploadFile.needUpload){
console.log("3.上传文件");// 分片上传文件// 确定分片的大小uploadChunk(xhrData.file,1, uploadFile);}}//上传文件分片functionuploadChunk(file, index, uploadFile){var chunkSize =1024*1024*10;//10mbvar chunkTotal = Math.ceil(file.size / chunkSize);if(index <= chunkTotal){// 根据是否暂停，确定是否继续上传// console.log("4.上传分片");var startTime =newDate().valueOf();var exit = uploadFile.chunkList.includes(index);// console.log("是否存在",exit);if(!exit){// console.log("3.3上传文件",uploadingStop);if(!uploadFile.uploadingStop){// 分片上传，同时计算进度条和上传速度// 已经上传的不在上传、// 上传完成后提示，上传成功// console.log("上传分片1",index);var form =newFormData();var start =(index -1)* chunkSize;let end =
index * chunkSize >= file.size ? file.size : index * chunkSize;let chunk = file.slice(start, end);// downloadBlob(chunk,file)// console.log("chunk",chunk);
form.append("chunk", chunk);
form.append("index", index);
form.append("chunkTotal", chunkTotal);
form.append("chunkSize", chunkSize);
form.append("md5", uploadFile.md5);
form.append("fileSize", file.size);
form.append("fileName", file.name);// console.log("上传分片", index);request({url:"/upload/chunk",method:"post",data: form,}).then((res)=>{var endTime =newDate().valueOf();var timeDif =(endTime - startTime)/1000;// console.log("上传文件大小",formatSize(chunkSize));// console.log("耗时",timeDif);// console.log("then",index);// uploadSpeed = (chunkSize/(1024*1024)) / timeDif +" M / s"
uploadFile.uploadSpeed =(10/ timeDif).toFixed(1)+" M/s";// console.log(res.data.data);// console.log("f2",uploadFile);
uploadFile.chunkList.push(index);// console.log("f3",uploadFile);
uploadFile.uploadPercentage =parseInt((uploadFile.chunkList.length / chunkTotal)*100);// console.log("上传进度",uploadFile.uploadPercentage);if(index == chunkTotal){
emitter.emit("reloadFileList");}uploadChunk(file, index +1, uploadFile);});}}else{
uploadFile.uploadPercentage =parseInt((uploadFile.chunkList.length / chunkTotal)*100);uploadChunk(file, index +1, uploadFile);}// }}}</script><stylescoped>.main{margin-top: 40px;margin-bottom: 40px;}.uploading{padding-top: 27px;}.progress{/* width: 700px; */display: flex;}.uploading .parse{display: flex;}.parse .el-progress{/* font-size: 18px; */width: 590px;}.progress .el-progress{/* font-size: 18px; */width: 590px;}.uploading .fileName{font-size: 17px;margin-right: 40px;margin-left: 80px;/* width: 80px; */}.uploading .fileSize{font-size: 17px;/* width: 80px; */}.progress .uploadSpeed{font-size: 17px;margin-left: 5px;padding-left: 5px;padding-right: 10px;}</style>
后端代码
packagecom.cugb.bigfileupload.controller;importcom.cugb.bigfileupload.bean.FilePO;importcom.cugb.bigfileupload.bean.Result;importcom.cugb.bigfileupload.servie.ChunkService;importcom.cugb.bigfileupload.servie.FileService;importorg.slf4j.Logger;importorg.slf4j.LoggerFactory;importorg.springframework.beans.factory.annotation.Autowired;importorg.springframework.beans.factory.annotation.Value;importorg.springframework.web.bind.annotation.*;importorg.springframework.web.multipart.MultipartFile;importjava.util.HashMap;importjava.util.List;importjava.util.Map;importjava.util.Objects;@RestController@CrossOriginpublicclassFileController{Logger logger =LoggerFactory.getLogger(getClass());@Value("${file.path}")privateString filePath;@AutowiredprivateFileService fileService;@AutowiredprivateChunkService chunkService;@GetMapping("/check")publicResultcheckFile(@RequestParam("md5")String md5){
logger.info("检查MD5:"+md5);//首先检查是否有完整的文件Boolean isUploaded = fileService.selectFileByMd5(md5);Map<String,Object> data =newHashMap<>();
data.put("isUploaded",isUploaded);//如果有，就返回秒传if(isUploaded){returnnewResult(201,"文件已经秒传",data);}//如果没有，就查找分片信息，并返回给前端List<Integer> chunkList = chunkService.selectChunkListByMd5(md5);
data.put("chunkList",chunkList);returnnewResult(201,"",data);}@PostMapping("/upload/chunk")publicResultuploadChunk(@RequestParam("chunk")MultipartFile chunk,@RequestParam("md5")String md5,@RequestParam("index")Integer index,@RequestParam("chunkTotal")Integer chunkTotal,@RequestParam("fileSize")Long fileSize,@RequestParam("fileName")String fileName,@RequestParam("chunkSize")Long chunkSize
){String[] splits = fileName.split("\\.");String type = splits[splits.length-1];String resultFileName = filePath+md5+"."+type;
chunkService.saveChunk(chunk,md5,index,chunkSize,resultFileName);
logger.info("上传分片："+index +" ,"+chunkTotal+","+fileName+","+resultFileName);if(Objects.equals(index, chunkTotal)){FilePO filePO =newFilePO(fileName, md5, fileSize);
fileService.addFile(filePO);
chunkService.deleteChunkByMd5(md5);returnnewResult(200,"文件上传成功",index);}else{returnnewResult(201,"分片上传成功",index);}}@GetMapping("/fileList")publicResultgetFileList(){
logger.info("查询文件列表");List<FilePO> fileList = fileService.selectFileList();returnnewResult(201,"文件列表查询成功",fileList);}}
4.2 文件下载模块
文件下载的流程图如图8所示，顺序图如图9所示
文件下载是首先，前端向后端发送分片下载的请求，请求的responseType设为blob(Binary large Object) ,然后后端通过RandomAccess类读取指定字节的内容，再写入到响应的文件流中。
浏览器前端的请求的分片数据，会暂时保存在“C:\Users\用户名\AppData\Local\Microsoft\Edge\User Data\Default\blob_storage\”中，（请确保c盘有足够的空间），当所有分片下载完成，会合并成一个大文件（很快），分片不是放在内存中，所以不用担心文件太大是不是不行。
，
刷新浏览器，也会删除已经下载好的分片
当前端请求了所有的文件分片之后，再把所有的blob合并成一个blob
if(index == chunkTotal){var resBlob =newBlob(file.blobList,{type:"application/octet-stream",});// console.log("resb", resBlob);let url = window.URL.createObjectURL(resBlob);// 将获取的文件转化为blob格式let a = document.createElement("a");// 此处向下是打开一个储存位置
a.style.display ="none";
a.href = url;// 下面两行是自己项目需要的处理，总之就是得到下载的文件名（加后缀）即可var fileName = file.name;
a.setAttribute("download", fileName);
document.body.appendChild(a);
a.click();//点击下载
document.body.removeChild(a);// 下载完成移除元素
window.URL.revokeObjectURL(url);// 释放掉blob对象}
图9文件上传顺序图
前端代码
<template><divclass="main"><divclass="fileList"><divclass="title">
文件列表
<!-- <hr> --></div><el-table:data="fileList"borderstyle="width: 360px"><el-table-columnprop="name"label="文件名"width="150"></el-table-column><el-table-columnprop="size"label="文件大小"width="110"><template#default="scope">
{{ formatSize(scope.row) }}
</template></el-table-column><el-table-columnprop=""label="操作"width="100"><template#default="scope"><el-buttonsize="small"type="primary"@click="downloadFile(scope.row)">下载</el-button></template></el-table-column></el-table></div><divclass="downloadList"><el-dividercontent-position="left">下载列表</el-divider><divv-for="file in downloadingFileList"><divclass="downloading"><spanclass="fileName">{{ file.name }}</span><spanclass="fileSize">{{ formatSize(file) }}</span><spanclass="downloadSpeed">{{ file.downloadSpeed }}</span><divclass="progress"><span>下载进度：</span><el-progress:text-inside="true":stroke-width="16":percentage="file.downloadPersentage"></el-progress><el-buttoncirclelink@click="changeDownloadStop(file)"><el-iconsize="20"v-if="file.downloadingStop == false"><VideoPause/></el-icon><el-iconsize="20"v-else><VideoPlay/></el-icon></el-button></div></div></div></div></div></template><scriptsetup>import axios from"axios";import{ ref, reactive, getCurrentInstance }from"vue";import emitter from"../utils/eventBus.js";import{ VideoPause, VideoPlay }from"@element-plus/icons-vue";const{ appContext }=getCurrentInstance();const request = appContext.config.globalProperties.request;var fileList =reactive([]);var downloadingFileList =ref([]);//上传文件之后，重新加载文件列表
emitter.on("reloadFileList",()=>{load();});functionload(){
fileList.length =0;request({url:"/fileList",method:"get",}).then((res)=>{// console.log("res", res.data.data);
fileList.push(...res.data.data);});}load();//换算文件的大小单位functionformatSize(file){//console.log("size",file.size);var size = file.size;var unit;var units =[" B"," K"," M"," G"];var pointLength =2;while((unit = units.shift())&& size >1024){
size = size /1024;}return((unit ==="B"? size
: size.toFixed(pointLength ===undefined?2: pointLength))+ unit
);}//点击暂停下载functionchangeDownloadStop(file){
file.downloadingStop =!file.downloadingStop;if(!file.downloadingStop){
console.log("开始。。");downloadChunk(1, file);}}//点击下载文件functiondownloadFile(file){// console.log("下载", file);
file.downloadingStop =false;
file.downloadSpeed ="0 M/s";
file.downloadPersentage =0;
file.blobList =[];
file.chunkList =[];
downloadingFileList.value.push(file);downloadChunk(1, file);}//点击下载文件分片functiondownloadChunk(index, file){var chunkSize =1024*1024*5;var chunkTotal = Math.ceil(file.size / chunkSize);if(index <= chunkTotal){// console.log("下载进度",index);var exit = file.chunkList.includes(index);
console.log("存在", exit);if(!exit){if(!file.downloadingStop){var formData =newFormData();
formData.append("fileName", file.name);
formData.append("md5", file.md5);
formData.append("chunkSize", chunkSize);
formData.append("index", index);
formData.append("chunkTotal", chunkTotal);if(index * chunkSize >= file.size){
chunkSize = file.size -(index -1)* chunkSize;
formData.set("chunkSize", chunkSize);}var startTime =newDate().valueOf();axios({url:"http://localhost:9001/download",method:"post",data: formData,responseType:"blob",timeout:50000,}).then((res)=>{
file.chunkList.push(index);var endTime =newDate().valueOf();var timeDif =(endTime - startTime)/1000;
file.downloadSpeed =(5/ timeDif).toFixed(1)+" M/s";//todo
file.downloadPersentage =parseInt((index / chunkTotal)*100);// var chunk = res.data.data.chunk// const blob = new Blob([res.data]);const blob = res.data;
file.blobList.push(blob);// console.log("res", blobList);if(index == chunkTotal){var resBlob =newBlob(file.blobList,{type:"application/octet-stream",});// console.log("resb", resBlob);let url = window.URL.createObjectURL(resBlob);// 将获取的文件转化为blob格式let a = document.createElement("a");// 此处向下是打开一个储存位置
a.style.display ="none";
a.href = url;// 下面两行是自己项目需要的处理，总之就是得到下载的文件名（加后缀）即可var fileName = file.name;
a.setAttribute("download", fileName);
document.body.appendChild(a);
a.click();//点击下载
document.body.removeChild(a);// 下载完成移除元素
window.URL.revokeObjectURL(url);// 释放掉blob对象}downloadChunk(index +1, file);});}}else{
file.downloadPersentage =parseInt((index / chunkTotal)*100);downloadChunk(index +1, file);}}}</script><stylescoped>.main{display: flex;}.fileList{width: 400px;}.downloadList{width: 450px;}.title{margin-top: 5px;margin-bottom: 5px;}.downloading{margin-top: 10px;}.downloading .fileName{margin-left: 76px;margin-right: 30px;}.downloading .fileSize{/* margin-left: 70px; */margin-right: 30px;}.downloading .progress{display: flex;}.progress .el-progress{/* font-size: 18px; */width: 310px;}</style>
后端代码
packagecom.cugb.bigfileupload.controller;importcom.cugb.bigfileupload.servie.ChunkService;importorg.slf4j.Logger;importorg.slf4j.LoggerFactory;importorg.springframework.beans.factory.annotation.Autowired;importorg.springframework.beans.factory.annotation.Value;importorg.springframework.stereotype.Controller;importorg.springframework.web.bind.annotation.*;importjavax.servlet.ServletOutputStream;importjavax.servlet.http.HttpServletResponse;importjava.io.File;importjava.io.IOException;importjava.util.Objects;@Controller@CrossOriginpublicclassDownLoadController{Logger logger =LoggerFactory.getLogger(getClass());@Value("${file.path}")privateString filePath;@AutowiredprivateChunkService chunkService;@PostMapping("/download")publicvoiddownload(@RequestParam("md5")String md5,@RequestParam("fileName")String fileName,@RequestParam("chunkSize")Integer chunkSize,@RequestParam("chunkTotal")Integer chunkTotal,@RequestParam("index")Integer index,HttpServletResponse response){String[] splits = fileName.split("\\.");String type = splits[splits.length -1];String resultFileName = filePath + md5 +"."+ type;File resultFile =newFile(resultFileName);long offset =(long) chunkSize *(index -1);if(Objects.equals(index, chunkTotal)){
offset = resultFile.length()-chunkSize;}byte[] chunk = chunkService.getChunk(index, chunkSize, resultFileName,offset);
logger.info("下载文件分片"+ resultFileName +","+ index +","+ chunkSize +","+ chunk.length+","+offset);// response.addHeader("Access-Control-Allow-Origin","Content-Disposition");
response.addHeader("Content-Disposition","attachment;filename="+ fileName);
response.addHeader("Content-Length",""+(chunk.length));
response.setHeader("filename", fileName);
response.setContentType("application/octet-stream");ServletOutputStream out =null;try{
out = response.getOutputStream();
out.write(chunk);
out.flush();
out.close();}catch(IOException e){
e.printStackTrace();}}}
4.3 数据库设计
4.3.1 概念结构设计
数据库设计只有俩个表，一个file表来记录已经完整上传的文件信息，一个chunk表用来记录还未上传完成的分片信息
5.1 大文件上传实现
上传页面如图13所示，有一个“选择上传文件”的按钮，下面是显示正在上传文件的列表
图13 上传页面首页
我们选择要上传的文件，确认上传，首先会显示解析进度，当解析完成后，就会开始上传，并显示上传进度和速度；同时，我们可以选择多个文件一同上传；在上传的同时我们还可以暂停上传。如图14所示
图14 上传文件中
当文件上传成功之后，就会弹窗提示文件上传成功。如图15所示
图15 文件上传成功
5.2 大文件下载实现
文件下载页面如图16所示，左边是可以下载文件的列表，右边是下载中的文件
当所有的分片下载完成后，前端会将所有的分片合并成一个文件。如图18所示
本文转载自: https://blog.csdn.net/weixin_50799082/article/details/128547482
版权归原作者 橙子1111 所有， 如有侵权，请联系我们删除。
版权归原作者 橙子1111 所有， 如有侵权，请联系我们删除。