SpringBoot集成fastdfs实现文件上传和下载

FastDFS是一个开源的轻量级分布式文件系统，它对文件进行管理，功能包括：文件存储、文件同步、文件访问（文件上传、文件下载）等，解决了大容量存储和负载均衡的问题。特别适合以文件为载体的在线服务，如相册网站、视频网站等等。
FastDFS为互联网量身定制，充分考虑了冗余备份、负载均衡、线性扩容等机制，并注重高可用、高性能等指标，使用FastDFS很容易搭建一套高性能的文件服务器集群提供文件上传、下载等服务。

Fastdfs介绍

简介

FastDFS服务端有两个角色：跟踪器（tracker）和存储节点（storage）。跟踪器主要做调度工作，在访问上起负载均衡的作用。
存储节点存储文件，完成文件管理的所有功能：就是这样的存储、同步和提供存取接口，FastDFS同时对文件的metadata进行管理。所谓文件的meta data就是文件的相关属性，以键值对（key value）方式表示，如：width=1024，其中的key为width，value为1024。文件metadata是文件属性列表，可以包含多个键值对。
跟踪器和存储节点都可以由一台或多台服务器构成。跟踪器和存储节点中的服务器均可以随时增加或下线而不会影响线上服务。其中跟踪器中的所有服务器都是对等的，可以根据服务器的压力情况随时增加或减少。
为了支持大容量，存储节点（服务器）采用了分卷（或分组）的组织方式。存储系统由一个或多个卷组成，卷与卷之间的文件是相互独立的，所有卷的文件容量累加就是整个存储系统中的文件容量。一个卷可以由一台或多台存储服务器组成，一个卷下的存储服务器中的文件都是相同的，卷中的多台存储服务器起到了冗余备份和负载均衡的作用。
在卷中增加服务器时，同步已有的文件由系统自动完成，同步完成后，系统自动将新增服务器切换到线上提供服务。
当存储空间不足或即将耗尽时，可以动态添加卷。只需要增加一台或多台服务器，并将它们配置为一个新的卷，这样就扩大了存储系统的容量。
FastDFS中的文件标识分为两个部分：卷名和文件名，二者缺一不可。

上传交互过程

1. client询问tracker上传到的storage，不需要附加参数；
2. tracker返回一台可用的storage；
3. client直接和storage通讯完成文件上传。

下载交互过程

1. client询问tracker下载文件的storage，参数为文件标识（卷名和文件名）；
2. tracker返回一台可用的storage；
3. client直接和storage通讯完成文件下载。
   需要说明的是，client为使用FastDFS服务的调用方，client也应该是一台服务器，它对tracker和storage的调用均为服务器间的调用。

搭建fastdfs测试环境

博主使用docker搭建的测试环境，所以相关资源都是docker相关的，如果没有使用过docker或者需要其他的搭建方式，请自行寻找并搭建。

相关文章

docker学习手册
docker-compose安装fastdfs

SpringBoot集成

Maven配置

<properties>
	<fastdfs.version>1.27.0.0</fastdfs.version>
	<tika.version>2.6.0</tika.version>
</properties>
<dependencies>
    <!--文件上传相关-->
    <dependency>
        <groupId>org.apache.tika</groupId>
        <artifactId>tika-core</artifactId>
        <version>${tika.version}</version>
    </dependency>
    <dependency>
        <groupId>net.oschina.zcx7878</groupId>
        <artifactId>fastdfs-client-java</artifactId>
        <version>${fastdfs.version}</version>
    </dependency>
</dependencies>

项目配置

fast_dfs_client.conf

connect_timeout=30
network_timeout=60
charset=UTF-8
http.tracker_http_port=5988
http.anti_steal_token=no
http.secret_key=
## 修改成自己的配置
tracker_server=192.168.11.182:22122

application-dev.yml

project:
  attachment:
    ## 该配置项是否为了上传文件将记录添加到数据库，方便下载
    download-base-uri: http://localhost:${server.port}/api/openApi/link/
    ## 文件后缀白名单集合(不支持通配符匹配，如果需要，请自行扩展)
    suffix-white-list:
      file: "text/plain,application/x-tika-ooxml,application/pdf"

AttachmentProperties.java

package com.pzx.platform.framework.conf;

import cn.hutool.core.util.StrUtil;
import lombok.Data;
import lombok.extern.slf4j.Slf4j;
import org.apache.tika.Tika;
import org.springframework.boot.context.properties.ConfigurationProperties;

import javax.annotation.Resource;
import java.util.Arrays;
import java.util.Map;

/**
 * @author JunPzx
 */
@Slf4j
@Data
@ConfigurationProperties("project.attachment")
public class AttachmentProperties {

    @Resource
    private Tika tika;

    /**
     * 后缀白名单
     */
    private Map<String, String> suffixWhiteList;

    /**
     * 下载基本uri
     */
    private String downloadBaseUri;


    /**
     * 白名单过滤后缀
     *
     * @param suffix 后缀
     * @param type   类型
     * @return boolean
     **/
    public boolean filterSuffixWhiteList(String suffix, String type) {
        log.debug(String.format("待检测的文件类型为：%s", suffix));
        if (StrUtil.isNotBlank(type)) {
            return Arrays.asList(suffixWhiteList.get(type).split(",")).contains(suffix);
        }
        return Arrays.asList(String.join(",", suffixWhiteList.values()).split(",")).contains(suffix);
    }

}

ToolsConfigurator.java

package com.pzx.platform.framework.conf;

import lombok.extern.slf4j.Slf4j;
import org.apache.tika.Tika;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
 * 工具配置器
 *
 * @author JunPzx
 * @version 1.0.0
 * @date 2022-07-01 14:30
 */
@Slf4j
@Configuration
public class ToolsConfigurator {

    @Bean
    public Tika obtainTika() {
        return new Tika();
    }
}

FastDfsConfigurator.java

package com.pzx.platform.framework.conf;

import lombok.extern.slf4j.Slf4j;
import org.csource.fastdfs.ClientGlobal;
import org.csource.fastdfs.StorageClient;
import org.csource.fastdfs.TrackerClient;
import org.csource.fastdfs.TrackerServer;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.Resource;

import java.io.IOException;

/**
 * fastDFS文件服务器配置
 *
 * @author JunPzx
 * @version 1.0.0
 * @since 2022-12-20 19:15:07
 */
@Configuration
@Slf4j
public class FastDfsConfigurator {

    @Value("classpath:fast_dfs_client.conf")
    private Resource ccs;

    @Bean
    public TrackerClient initTrackerClient() {
        try {
            ClientGlobal.init(ccs.getFilename());
            return new TrackerClient();
        } catch (Exception e) {
            log.error("FastDFS创建客户端失败");
            return null;
        }
    }

    @Bean
    public StorageClient initStorageClient(TrackerClient trackerClient) throws IOException {
        try {
            log.debug("tracker server:" + trackerClient.getConnection().getInetSocketAddress().getHostName() + ";port:" + trackerClient.getConnection().getInetSocketAddress().getPort());
            TrackerServer trackerServer = trackerClient.getConnection();
            return new StorageClient(trackerServer, null);
        } catch (Exception e) {
            log.error("FastDFS创建StorageClient失败");
            throw e;
        }

    }
}

OpenApiController.java

文件上传案例，其中的很多类我就不放上面的，核心逻辑都在，没放上来的都是一些边角，ISysAttachmentService是否系统附件服务，用于上传文件备案的，里面存储了附件的一些信息

package com.pzx.platform.mgt.api.controller;


import com.pzx.core.exception.ApplicationException;
import com.pzx.core.message.BusinessCodeEnum;
import com.pzx.platform.mgt.sys.entity.SysAttachment;
import com.pzx.platform.mgt.sys.service.ISysAttachmentService;
import io.swagger.annotations.Api;
import io.swagger.annotations.ApiOperation;
import io.swagger.annotations.ApiParam;
import org.csource.common.MyException;
import org.csource.fastdfs.StorageClient1;
import org.csource.fastdfs.TrackerClient;
import org.csource.fastdfs.TrackerServer;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import javax.annotation.Resource;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.nio.charset.StandardCharsets;

/**
 * 开放api控制器
 *
 * @author JunPzx
 * @version 1.0.0
 * @since 2022-12-20 18:35:48
 */
@RestController
@RequestMapping("/openApi")
@Api(value = "开放API接口", tags = {"开放API接口"})
public class OpenApiController {

    @Resource
    private ISysAttachmentService sysAttachmentService;

    @Resource
    TrackerClient trackerClient;

    /**
     * 初始化存储client1
     *
     * @param trackerClient 跟踪客户
     * @return {@link StorageClient1}
     * @throws IOException ioexception
     */
    private StorageClient1 initStorageClient1(TrackerClient trackerClient) throws IOException {
        TrackerServer trackerServer = trackerClient.getConnection();
        return new StorageClient1(trackerServer, null);
    }

    /**
     * 附件链接
     *
     * @param attachmentId 附件标识
     * @return {@link ResponseEntity }
     **/
    @ApiOperation(value = "附件链接", notes = "附件链接")
    @GetMapping("/link/{attachmentId}")
    public ResponseEntity<byte[]> link(@PathVariable String attachmentId
            , @ApiParam(hidden = true) HttpServletResponse response
            , @ApiParam(hidden = true) HttpServletRequest request) throws IOException {
        SysAttachment sysAttachment = sysAttachmentService.getById(attachmentId);
        if (sysAttachment == null) {
            throw new ApplicationException(BusinessCodeEnum.SYS_ATTACHMENT_NOT_EXIST);
        }
        try {
            byte[] bytes = initStorageClient1(trackerClient).download_file1(sysAttachment.getFilePath());
            response.setContentType(MediaType.parseMediaType(sysAttachment.getMetadata()).toString());
            String userAgent = request.getHeader("User-Agent");
            String formFileName = sysAttachment.getOriFileName();
            // 针对IE或者以IE为内核的浏览器：
            if (userAgent.contains("MSIE") || userAgent.contains("Trident")) {
                formFileName = java.net.URLEncoder.encode(formFileName, "UTF-8");
            } else {
                // 非IE浏览器的处理：
                formFileName = new String(formFileName.getBytes(StandardCharsets.UTF_8), StandardCharsets.ISO_8859_1);
            }
            response.addHeader("Content-Disposition", String.format("attachment; filename=\"%s\"", formFileName));
            return ResponseEntity.ok(bytes);
        } catch (IOException | ApplicationException | MyException e) {
            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR)
                    .build();
        }
    }
}

SysAttachmentController.java

下载案例

package com.xatali.platform.mgt.sys.controller;


import cn.hutool.core.bean.BeanUtil;
import cn.hutool.core.convert.Convert;
import cn.hutool.core.io.FileUtil;
import cn.hutool.core.util.StrUtil;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;
import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
import com.pzx.core.message.BusinessCodeEnum;
import com.pzx.core.utils.TimestampUUIDGenerator;
import com.pzx.platform.framework.conf.AttachmentProperties;
import com.pzx.platform.mgt.sys.dto.SysAttachmentDTO;
import com.pzx.platform.mgt.sys.entity.SysAttachment;
import com.pzx.platform.mgt.sys.service.ISysAttachmentService;
import io.swagger.annotations.Api;
import io.swagger.annotations.ApiOperation;
import io.swagger.annotations.ApiParam;
import lombok.extern.slf4j.Slf4j;
import org.apache.tika.Tika;
import org.csource.common.MyException;
import org.csource.common.NameValuePair;
import org.csource.fastdfs.StorageClient1;
import org.csource.fastdfs.TrackerClient;
import org.csource.fastdfs.TrackerServer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.MediaType;
import org.springframework.web.bind.annotation.*;
import com.pzx.core.annotation.OperationLog;
import com.pzx.core.domain.Result;
import com.pzx.core.enumerable.OperationEnum;
import org.springframework.web.multipart.MultipartFile;

import javax.annotation.Resource;
import java.io.IOException;
import java.util.*;
import java.util.concurrent.atomic.AtomicInteger;

/**
 * 系统附件表(SysAttachment)控制层
 *
 * @author JunPzx
 * @since 2022-12-20 11:08:56
 */
@RestController
@RequestMapping("sysAttachment")
@Api(tags = "系统附件表管理")
@Slf4j
public class SysAttachmentController {
    /**
     * 服务对象
     */
    @Resource
    private ISysAttachmentService sysAttachmentService;

    @Resource
    private Tika tika;

    @Autowired
    private AttachmentProperties attachmentProperties;

    @Resource
    private TimestampUUIDGenerator timestampUUIDGenerator;

    @Resource
    private TrackerClient trackerClient;


    /**
     * 初始化存储client1
     *
     * @param trackerClient 跟踪客户
     * @return {@link StorageClient1}
     * @throws IOException ioexception
     */
    private StorageClient1 initStorageClient1(TrackerClient trackerClient) throws IOException {
        TrackerServer trackerServer = trackerClient.getConnection();
        return new StorageClient1(trackerServer, null);
    }

    /**
     * 多上传
     *
     * @param file 多部分文件
     * @return {@link Result<SysAttachment> }
     **/
    @ApiOperation(value = "上传附件", notes = "上传附件", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
    @PostMapping("/multiUpload")
    @OperationLog(operation = OperationEnum.UPLOAD)
    public Result<List<Map<String, String>>> multiUpload(
            @ApiParam("待上传文件")
            @RequestParam(value = "file") MultipartFile[] file) {
        List<MultipartFile> files = Arrays.asList(file);
        List<Map<String, String>> uploadResult = new ArrayList<>();
        if (files.isEmpty()) {
            return Result.error("上传失败，请选择文件");
        }
        AtomicInteger uploadStep = new AtomicInteger(0);
        files.parallelStream().forEach(t -> {
            // 后缀白名单过滤
            try {
                String metadata = tika.detect(t.getInputStream());
                if (!attachmentProperties.filterSuffixWhiteList(metadata, "file")) {
                    log.warn(String.format("%s文件的元数据类型%s不在安全名单内", t.getOriginalFilename(), metadata));
                    return;
                }
                if (t.isEmpty()) {
                    log.warn("不允许上传空文件");
                    return;
                }
                String fileName = timestampUUIDGenerator.generate().toUpperCase() + "." + FileUtil.getSuffix(t.getOriginalFilename());
                NameValuePair[] metaList = new NameValuePair[3];
                metaList[0] = new NameValuePair("fileName", fileName);
                metaList[1] = new NameValuePair("fileExt", FileUtil.getSuffix(t.getOriginalFilename()));
                metaList[2] = new NameValuePair("fileSize", String.valueOf(t.getSize()));
                log.info("tracker server:" + trackerClient.getConnection().getInetSocketAddress().getHostName() + ";port:" + trackerClient.getConnection().getInetSocketAddress().getPort());
                String path = initStorageClient1(trackerClient).upload_file1(t.getBytes(), FileUtil.getSuffix(t.getOriginalFilename()), metaList);
                if (!StrUtil.isEmpty(path)) {
                    SysAttachment sysAttachment = SysAttachment.builder()
                            .fileName(fileName)
                            .filePath(path)
                            .fileSize(t.getSize())
                            .oriFileName(t.getOriginalFilename())
                            .metadata(metadata)
                            .build();
                    // 插入附件记录表
                    sysAttachmentService.save(sysAttachment);
                    Map<String, String> uploadStepResult = new HashMap<>(8);
                    uploadStepResult.put("originalFilename", t.getOriginalFilename());
                    uploadStepResult.put("uri", attachmentProperties.getDownloadBaseUri() + sysAttachment.getId());
                    uploadStepResult.put("attachmentId", sysAttachment.getId().toString());
                    uploadResult.add(uploadStepResult);
                    log.info("上传第" + uploadStep.incrementAndGet() + "个文件成功");
                }
            } catch (IOException | MyException e) {
                log.error("文件上传异常", e);
            }
        });
        if (uploadStep.get() != files.size()) {
            return Result.error(BusinessCodeEnum.SYS_ATTACHMENT_PART_UPLOAD_FAIL, uploadResult);
        }
        return Result.success(uploadResult);
    }

}

上传和下载

上传

下载