Skip to content

feat(fs): add pipelined multipart upload#2723

Open
thetapilla wants to merge 7 commits into
OpenListTeam:mainfrom
thetapilla:feat/multipart-upload
Open

feat(fs): add pipelined multipart upload#2723
thetapilla wants to merge 7 commits into
OpenListTeam:mainfrom
thetapilla:feat/multipart-upload

Conversation

@thetapilla

@thetapilla thetapilla commented Jul 5, 2026

Copy link
Copy Markdown

Summary / 摘要

实现网页端分块上传(multipart upload),解决经 CDN 反代(如 Cloudflare 免费计划 100MB 请求体上限)时无法上传大文件的问题,并为上传链路补上断点续传与失败重试能力。

用户可感知的变化:

  • 新增第四种上传方式 Multipart :客户端按分片并发上传,单个请求体可控制在 CDN 限制之内(分片大小可配,默认 10MB)
  • 断点续传:上传中断后(网络断开、刷新页面)重新发起同一文件的上传会自动跳过已接收的分片继续
  • 失败重试:整体上传失败后会话元数据保留 30 分钟,客户端从 chunk 0 重灌即可重试,无需重新走完整流程
  • 秒传:勾选"尝试秒传"时哈希随会话初始化先行到达驱动,秒传命中则后续分片全部免传(对应 feat(fs): implement client-side multipart upload with File-Path header #1877 评审意见中的期望)
  • 管理端新增两个设置:multipart_enabled(功能开关)、multipart_chunk_size(分片大小 MB,保存时校验 1–90)

实现要点:

  • 流水线而非收齐再传:会话初始化即调用 op.Put 启动驱动上传,客户端→服务端与服务端→存储两段并行,分别占用服务器下行/上行带宽
  • 环形窗口重组:乱序并发到达的分片经环形暂存文件重组为顺序流,每会话磁盘占用封顶为 8×分片大小(默认约 80MB),分片被驱动消费后即释放;不做任何全量落盘
  • 零驱动改动:驱动侧拿到的仍是标准的顺序 FileStreamer,与现有 /fs/put 直传对驱动完全同构,87 个驱动无需任何适配(已在本地/S3/WebDAV/百度/123 五种驱动形态上实测)
  • 浏览器友好的流控:窗口满时分片请求在服务端有界停靠(上限 10s,远低于 CDN 请求时限)等待槽位而非立即拒绝;所有响应路径先排干请求体再回包——否则浏览器会把提前响应视为连接错误
  • 数据安全:断点续传要求哈希证明文件身份(防止同路径同大小不同内容的静默混合);重灌时逐片 CRC32 校验拦截"两次重试间文件内容变化"
  • 回收完整:成功/中止/失败/被取代/过期/重启孤儿六类终止路径均验证无分片数据残留,孤儿暂存文件在服务启动时清扫

关于与 As-Task 的关系:本 PR 中 multipart 有意不支持"添加为任务"。As-Task 的语义是"先收完、入队、稍后传",必然要求文件全量落盘,与流水线形态(收片即传、磁盘占用窗口封顶、秒传免传后续分片)在根本上互斥;且 multipart 会话自身已具备任务队列的核心价值(进度可查、失败可断点重试)。若社区确有"multipart + As-Task"的需求(全量落盘后再推远端,形态接近 #1877 的 store-and-forward 方案),可以作为后续独立 PR 在本 PR 的会话框架上补充,实现难度低于本 PR。

兼容性说明:不改动任何既有端点与驱动接口;新增 /api/fs/multipart/{init,chunk,complete,status,abort} 五个端点(header 风格与 /fs/put 对齐);新增两个 PUBLIC 设置项;会话仅存内存(与 Upload 任务不持久化的既有决策对齐),重启后未完成会话失效、暂存自动清扫。

  • This PR has breaking changes.
    / 此 PR 包含破坏性变更。
  • This PR changes public API, config, storage format, or migration behavior.
    / 此 PR 修改了公开 API、配置、存储格式或迁移行为。
  • This PR requires corresponding changes in related repositories.
    / 此 PR 需要关联仓库同步修改。

Related repository PRs / 关联仓库 PR:

Related Issues / 关联 Issue

Relates to #460

说明:#460 同时提出了网页端与 WebDAV 两个方向的分块上传,本 PR 覆盖网页端;WebDAV 分块(如 Nextcloud 兼容协议)不在本 PR 范围内,但服务端重组器与协议无关,后续可将 WebDAV/Nextcloud 入口作为它的另一个前端接入。

致谢:感谢 #1877@dnslin)对该方向的探索,以及 @KirCute 在该 PR 评审中提出的两点要求——"开始上传时即构建 FileStream 调用驱动(两段带宽并行)"与"首个请求携带文件哈希触发秒传、命中后免传后续分片"。本 PR 为独立的从零实现,上述两点均为本设计的出发点并已实现(哈希放在 init 请求而非首个分片,语义等价且更早:秒传判定可在任何分片到达前完成)。

Testing / 测试

测试平台:macOS arm64,Go 1.26.4。

  • go test ./...
    本机环境因缺少 macFUSE 头文件无法构建 internal/fuse(存量平台问题,与本 PR 无关),故未跑全仓测试;实际执行为下述包级测试与主二进制构建。
  • Manual test / 手动测试:

自动化测试:

  • go test -race -count=1 ./internal/multipart/:32 个用例全部通过,覆盖乱序/并发写入、背压停靠与唤醒、幂等重发、秒传与在途分片的竞态(含确定性竞态构造与变异验证)、窗口关闭与完成裁决之间的间隙、CRC 重灌一致性、会话状态机全迁移、GC 过期与阻塞读者唤醒
  • gofmt -l 对全部改动文件零违规;go build . 主二进制构建通过
  • 本地 e2e 脚本(local 驱动,59 项断言):100MB 乱序并发上传后哈希比对、断点续传、只读目录故障注入→修复→重灌→成功、abort 清理(含目标端无部分文件)、429 背压语义、设置门禁与非法值拒绝、无哈希重灌的数据混合防护
  • 资源回收审计:成功/中止/可恢复失败/重灌/被新上传取代/服务重启孤儿清扫,全部验证无暂存残留,并含全局空目录断言
  • 浏览器行为模拟压测(全局 6 连接上限、发送异常视为响应不可读):3×90MB 并发上传至限速 3MB/s 的存储,零错误、零误报,总耗时 90s(等于限速理论下限)

跨驱动实测:

  • MinIO S3(分片会话型驱动):200MB 上传+下载回读哈希一致;另测每片间隔 3s 的慢速客户端,会话空闲容忍正常
  • WebDAV 回环(单请求流式驱动):30MB 上传,落盘字节一致
  • 百度网盘真实账号(先落盘算哈希的 spool 型驱动):20MB 正常传输链路通过;秒传接口被百度侧拒绝(errno 31500,第三方应用限制)时回退为正常上传,数据完整
  • 123 云盘真实账号:秒传命中实录——同内容带 MD5 重传 0.8s 完成(对照全量传输 10.4s),服务端到存储零流量

浏览器手动验证(配合前端配套分支联调):覆盖已有文件 × 尝试秒传四种组合、刷新页面后重拖续传、3×3 大文件并发上传、失败行重试按钮断点续传、管理端分片大小非法值拒绝提示。

Checklist / 检查清单

  • I have read CONTRIBUTING.
    / 我已阅读 CONTRIBUTING
  • I confirm this contribution follows the repository license, contribution policy, and code of conduct.
    / 我确认此贡献符合仓库许可证、贡献规范和行为准则。
  • I have formatted the changed code with gofmt, go fmt, or prettier where applicable.
    / 我已按适用情况使用 gofmtgo fmtprettier 格式化变更代码。
  • I have requested review from relevant maintainers or code owners where applicable.
    / 我已在适用情况下请求相关维护者或代码所有者审查。

AI Disclosure / AI 使用声明

  • This PR includes AI-assisted content.
    / 此 PR 包含 AI 辅助内容。

Tools used / 使用工具:

  • ChatGPT
  • Codex
  • GitHub Copilot
  • Claude
  • Gemini
  • Other (please specify) / 其他(请注明):

Usage scope / 使用范围:

  • Code generation / 代码生成

  • Refactoring / 重构

  • Documentation / 文档

  • Tests / 测试

  • Translation / 翻译

  • Review assistance / 审查辅助

  • I have reviewed and validated all AI-assisted content included in this PR.
    / 我已审核并验证此 PR 中的所有 AI 辅助内容。

  • I have ensured that all AI-assisted commits include Co-Authored-By attribution.
    / 我已确保所有 AI 辅助提交都包含 Co-Authored-By 归属信息。

  • I can reproduce all AI-assisted content included in this PR without any AI tools.
    / 我可以在没有任何 AI 工具的情况下重现此 PR 中包含的所有 AI 辅助内容。

- Reassemble concurrently uploaded chunks into a sequential stream through a ring file, bounding disk usage to slots*chunkSize per session
- Park writers up to a deadline when their slot is busy instead of rejecting instantly, so flow control does not surface as connection errors in browsers
- Record a per-chunk CRC32 table for re-fill verification and keep it readable after close
- Propagate cancellation to blocked readers via CloseWithError so drivers treat aborts like canceled requests
- Cover ordering, backpressure, idempotent resends and close/abort wake-ups with race-enabled tests
- Start the driver upload at session init over a sequential stream backed by the window, so client-to-server and server-to-storage transfers run concurrently
- Attach client-provided hashes to the stream so drivers can attempt rapid upload before any chunk arrives, and absorb chunks racing pipeline completion idempotently
- Keep only metadata and chunk CRCs after a failed attempt: re-sending chunk 0 re-fills a fresh window, and content changes between attempts are rejected
- Resume receiving sessions only when client hashes prove the same file; failed_retriable sessions resume unconditionally
- Reclaim sessions with a sliding-TTL GC and sweep orphaned ring files at startup
- Cover the state machine with race-enabled tests over a stubbed storage layer
- Add multipart_enabled and multipart_chunk_size (MB) as public traffic settings
- Validate the chunk size on save (integer within 1-90) via the setting item hook
- Add /api/fs/multipart init/chunk/complete/status/abort endpoints with headers aligned with /fs/put
- Gate init behind the FsUp permission checks and reuse the client upload rate limiter for chunk uploads
- Drain the request body before answering chunk requests on every path, so browsers do not see early responses as network errors
- Start the multipart session GC when the router is initialized to reclaim ring files orphaned by a previous run
Comment thread server/router.go
Comment on lines +217 to +222
multipart := g.Group("/multipart")
multipart.POST("/init", middlewares.FsUp, handles.MultipartInit)
multipart.PUT("/chunk", uploadLimiter, handles.MultipartChunk)
multipart.POST("/complete", handles.MultipartComplete)
multipart.GET("/status", handles.MultipartStatus)
multipart.POST("/abort", handles.MultipartAbort)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

麻烦在GitHub上Comment一下新增的API文档,方便后面加到Apifox里面。

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wip

Comment thread internal/multipart/session.go Outdated
Comment thread internal/multipart/session.go Outdated
Comment thread internal/multipart/session.go Outdated
Comment thread internal/op/hook.go
Comment on lines +88 to +98
conf.MultipartChunkSize: func(item *model.SettingItem) error {
size, err := strconv.Atoi(strings.TrimSpace(item.Value))
if err != nil || size < 1 || size > 90 {
// deliberately a plain error: SaveSettings formats hook errors
// with %+v, which would dump a full stack trace into the UI
// notification for stack-carrying errors
return fmt.Errorf("multipart chunk size must be an integer between 1 and 90 (MB), got %q", item.Value)
}
item.Value = strconv.Itoa(size)
return nil
},

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

个人认为OpenList作为一个自部署项目,不应该对这个分片大小做严格的限制。让用户自己决定分片大小,框架只需要保证合理性即可。

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

同意这个理念,已在 b638510 调整:

  • 管理端设置:校验放宽为"≥1 的整数",设置项 Help 文案同步更新;
  • 其他:init 请求的 X-Chunk-Size header 是客户端可控的;服务端每会话缓冲 8 个分片的窗口,如果对客户端建议值也不设上限,任意登录用户可以在 init 时索要超大分片,直接放大服务端磁盘占用。因此 handler 将客户端建议值卡到 [1MB, 管理员设置值],即客户端不能越过管理员定的块大小。

- The overwrite flag was stored on the session but never read: the handler
  performs the pre-check and op.Put owns the overwrite semantics
- Validate the setting as a positive integer only; self-hosted admins decide
  the ceiling themselves instead of an arbitrary 90MB cap
- Keep clamping the client-suggested X-Chunk-Size to the admin value: the
  server buffers a window of 8 chunks per session, so an unbounded client
  suggestion would translate directly into server-side disk usage
@thetapilla thetapilla force-pushed the feat/multipart-upload branch from 60cd884 to b638510 Compare July 11, 2026 15:14
- Fold the two-step clamp into one branch: the ceiling is already floored,
  so a client suggestion just lowers the size with a 1MB floor
@thetapilla

thetapilla commented Jul 12, 2026

Copy link
Copy Markdown
Author

Multipart API 文档

此处附上 OpenAPI 3.0
openapi: 3.0.3
info:
  title: OpenList Multipart Upload API
  version: "1.0"
  description: >-
    Chunked, resumable, pipelined upload. Every endpoint requires a logged-in
    user (Authorization header). The HTTP status is always 200: the status
    codes documented under Responses refer to the "code" field of the JSON
    envelope {code, message, data}.
security:
  - bearerAuth: []
paths:
  /api/fs/multipart/init:
    post:
      summary: Init multipart upload
      description: >-
        Create an upload session and immediately start the server-to-driver
        upload pipeline, or resume an existing one. Session reuse rules (same
        user + path + size): a live "receiving" session is resumed only when
        the supplied file hashes match the ones it was created with — without
        hashes, path+size cannot prove file identity, so no resume; a
        "failed_retriable" session is always reused (its buffered data was
        already discarded, refill is guarded by per-chunk CRC); any other
        existing session is superseded and cleaned up. Supplied file hashes
        travel ahead of the file data, so drivers with server-side
        deduplication can finish instantly (rapid upload) without receiving
        any chunk.
      parameters:
        - {name: File-Path, in: header, required: true, schema: {type: string}, example: "/uploads/movie.mkv", description: "URL-encoded destination path (same as /fs/put)"}
        - {name: X-File-Size, in: header, required: true, schema: {type: integer, format: int64, minimum: 1}, description: "file size in bytes; upload empty files via /fs/put"}
        - {name: X-Chunk-Size, in: header, required: false, schema: {type: integer, format: int64}, description: "suggested chunk bytes, clamped to [1MB, admin setting]; the effective value is returned as chunk_size"}
        - {name: Overwrite, in: header, required: false, schema: {type: boolean, default: true}, description: "when false, reject if the target already exists"}
        - {name: X-File-Md5, in: header, required: false, schema: {type: string}, description: "MD5 of the file; enables rapid upload and identifies the file for resume"}
        - {name: X-File-Sha1, in: header, required: false, schema: {type: string}, description: "SHA1 of the file; enables rapid upload and identifies the file for resume"}
        - {name: X-File-Sha256, in: header, required: false, schema: {type: string}, description: "SHA256 of the file; enables rapid upload and identifies the file for resume"}
        - {name: Last-Modified, in: header, required: false, schema: {type: integer, format: int64}, description: "file modification time (Unix timestamp in milliseconds)"}
        - {name: Content-Type, in: header, required: false, schema: {type: string}, description: "file MIME type, inferred from the extension when omitted"}
      responses:
        "200":
          x-apifox-name: Success
          description: "Session created or resumed; data is the session snapshot plus resumed"
          content:
            application/json:
              schema:
                allOf:
                  - $ref: "#/components/schemas/Envelope"
                  - type: object
                    properties:
                      data:
                        allOf:
                          - $ref: "#/components/schemas/SessionSnapshot"
                          - type: object
                            properties:
                              resumed: {type: boolean, description: "true when an existing session was reused; skip the chunk ranges listed in received"}
              example:
                code: 200
                message: success
                data:
                  upload_id: 3f2c9e2a-6c4b-4d2e-9a2b-1f0e8c7d5a41
                  state: receiving
                  attempt: 0
                  path: /uploads/movie.mkv
                  size: 734003200
                  chunk_size: 10485760
                  total_chunks: 70
                  received: [[0, 2]]
                  received_bytes: 31457280
                  frontier: 3
                  storage_progress: 4.29
                  resumed: true
        "400":
          x-apifox-name: Bad Request
          description: "Invalid File-Path, X-File-Size or X-Chunk-Size"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Envelope"}
              example:
                code: 400
                message: multipart upload requires a valid X-File-Size header
                data: null
        "401":
          x-apifox-name: Unauthorized
          description: "Not logged in or the token has expired"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Envelope"}
              example:
                code: 401
                message: token is expired
                data: null
        "403":
          x-apifox-name: Forbidden
          description: "Multipart upload disabled, no write permission on the destination, or Overwrite is false and the target already exists"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Envelope"}
              example:
                code: 403
                message: permission denied
                data: null
        "405":
          x-apifox-name: Upload Not Supported
          description: "The target storage does not support upload"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Envelope"}
              example:
                code: 405
                message: upload not supported
                data: null
        "500":
          x-apifox-name: Storage Error
          description: "Failed to resolve the target storage"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Envelope"}
              example:
                code: 500
                message: "storage not found; rawPath: /uploads"
                data: null
  /api/fs/multipart/chunk:
    put:
      summary: Upload one chunk
      description: >-
        Send one chunk as the raw request body. Chunks are idempotent and may
        be sent concurrently and out of order, but only within the receiving
        window [frontier, frontier+8), which slides forward as the driver
        consumes data. Every chunk except the last must be exactly chunk_size
        bytes; the last one is the remainder. When the window is full the
        request parks on the server for up to 10 seconds waiting for a free
        slot before answering 429. If the session already completed (e.g.
        rapid upload) the chunk is absorbed idempotently and the returned
        snapshot has state=completed — stop sending the remaining chunks. The
        server always consumes the request body before answering, so browsers
        never see an early response.
      parameters:
        - {name: X-Upload-Id, in: header, required: true, schema: {type: string}, description: "session id returned by init"}
        - {name: X-Chunk-Index, in: header, required: true, schema: {type: integer, minimum: 0}, description: "0-based chunk index"}
      requestBody:
        required: true
        content:
          application/octet-stream:
            schema: {type: string, format: binary}
      responses:
        "200":
          x-apifox-name: Success
          description: "Chunk stored (or absorbed idempotently); data is the latest session snapshot"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/SnapshotResp"}
              example:
                code: 200
                message: success
                data:
                  upload_id: 3f2c9e2a-6c4b-4d2e-9a2b-1f0e8c7d5a41
                  state: receiving
                  attempt: 0
                  path: /uploads/movie.mkv
                  size: 734003200
                  chunk_size: 10485760
                  total_chunks: 70
                  received: [[0, 11]]
                  received_bytes: 125829120
                  frontier: 9
                  storage_progress: 12.86
        "400":
          x-apifox-name: Bad Chunk or Failed Session
          description: "Invalid X-Chunk-Index, wrong chunk length, a refilled chunk differs from the previous attempt, or the session already failed (data carries state and error)"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/SnapshotResp"}
              example:
                code: 400
                message: "upload attempt failed, resend from chunk 0 to retry: context deadline exceeded"
                data:
                  upload_id: 3f2c9e2a-6c4b-4d2e-9a2b-1f0e8c7d5a41
                  state: failed_retriable
                  attempt: 0
                  path: /uploads/movie.mkv
                  size: 734003200
                  chunk_size: 10485760
                  total_chunks: 70
                  received: []
                  received_bytes: 0
                  frontier: 0
                  storage_progress: 0
                  error: context deadline exceeded
        "401":
          x-apifox-name: Unauthorized
          description: "Not logged in or the token has expired"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Envelope"}
              example:
                code: 401
                message: token is expired
                data: null
        "403":
          x-apifox-name: Not Owner
          description: "The session belongs to another user (data carries no session info)"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Envelope"}
              example:
                code: 403
                message: multipart upload session belongs to another user
                data: null
        "404":
          x-apifox-name: Session Not Found
          description: "Session not found: expired, reclaimed after completion, or the server restarted (data carries no session info)"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Envelope"}
              example:
                code: 404
                message: multipart upload session not found
                data: null
        "409":
          x-apifox-name: Chunk In Flight
          description: "The same chunk is being uploaded on another connection; retry shortly (data carries the latest snapshot)"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/SnapshotResp"}
              example:
                code: 409
                message: chunk is being uploaded by another request
                data:
                  upload_id: 3f2c9e2a-6c4b-4d2e-9a2b-1f0e8c7d5a41
                  state: receiving
                  attempt: 0
                  path: /uploads/movie.mkv
                  size: 734003200
                  chunk_size: 10485760
                  total_chunks: 70
                  received: [[0, 11]]
                  received_bytes: 125829120
                  frontier: 9
                  storage_progress: 12.86
        "429":
          x-apifox-name: Window Full
          description: "Receiving window still full after parking; flow control, not an error — resend the chunk after a short delay (data carries the latest snapshot)"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/SnapshotResp"}
              example:
                code: 429
                message: chunk is out of the receiving window
                data:
                  upload_id: 3f2c9e2a-6c4b-4d2e-9a2b-1f0e8c7d5a41
                  state: receiving
                  attempt: 0
                  path: /uploads/movie.mkv
                  size: 734003200
                  chunk_size: 10485760
                  total_chunks: 70
                  received: [[0, 16]]
                  received_bytes: 178257920
                  frontier: 9
                  storage_progress: 12.86
  /api/fs/multipart/complete:
    post:
      summary: Complete multipart upload
      description: >-
        Join point: blocks until the upload pipeline reaches a terminal state,
        mirroring how /fs/put only answers once the driver upload finished.
        Rejected immediately while chunks are still missing, so an incomplete
        upload never holds the connection open. On success the session is
        reclaimed right away. If a CDN cuts the connection early, poll /status
        until a terminal state.
      parameters:
        - {name: X-Upload-Id, in: header, required: true, schema: {type: string}, description: "session id returned by init"}
      responses:
        "200":
          x-apifox-name: Success
          description: "Upload finished; data.state is completed and the session is reclaimed"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/SnapshotResp"}
              example:
                code: 200
                message: success
                data:
                  upload_id: 3f2c9e2a-6c4b-4d2e-9a2b-1f0e8c7d5a41
                  state: completed
                  attempt: 0
                  path: /uploads/movie.mkv
                  size: 734003200
                  chunk_size: 10485760
                  total_chunks: 70
                  received: [[0, 69]]
                  received_bytes: 734003200
                  frontier: 70
                  storage_progress: 100
        "400":
          x-apifox-name: Incomplete or Failed
          description: "Chunks still missing (returned immediately), or the pipeline failed (data carries state and error)"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/SnapshotResp"}
              example:
                code: 400
                message: "cannot complete: 723517440 of 734003200 bytes received"
                data:
                  upload_id: 3f2c9e2a-6c4b-4d2e-9a2b-1f0e8c7d5a41
                  state: receiving
                  attempt: 0
                  path: /uploads/movie.mkv
                  size: 734003200
                  chunk_size: 10485760
                  total_chunks: 70
                  received: [[0, 65], [67, 69]]
                  received_bytes: 723517440
                  frontier: 66
                  storage_progress: 94.29
        "401":
          x-apifox-name: Unauthorized
          description: "Not logged in or the token has expired"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Envelope"}
              example:
                code: 401
                message: token is expired
                data: null
        "403":
          x-apifox-name: Not Owner
          description: "The session belongs to another user (data carries no session info)"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Envelope"}
              example:
                code: 403
                message: multipart upload session belongs to another user
                data: null
        "404":
          x-apifox-name: Session Not Found
          description: "Session not found: expired, reclaimed after completion, or the server restarted (data carries no session info)"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Envelope"}
              example:
                code: 404
                message: multipart upload session not found
                data: null
  /api/fs/multipart/status:
    get:
      summary: Query upload session
      description: >-
        Two valid parameter combinations: pass upload_id alone, or pass path
        and size together to discover a resumable live session of the current
        user (e.g. after a page refresh, or to poll the verdict when complete
        was cut off by a CDN). Each parameter is optional on its own, but one
        of the two combinations is required.
      parameters:
        - {name: upload_id, in: query, required: false, schema: {type: string}, description: "session id; use alone"}
        - {name: path, in: query, required: false, schema: {type: string}, description: "URL-encoded destination path; use together with size"}
        - {name: size, in: query, required: false, schema: {type: integer, format: int64}, description: "file size in bytes; use together with path"}
      responses:
        "200":
          x-apifox-name: Success
          description: "Session found; data is the latest session snapshot"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/SnapshotResp"}
              example:
                code: 200
                message: success
                data:
                  upload_id: 3f2c9e2a-6c4b-4d2e-9a2b-1f0e8c7d5a41
                  state: receiving
                  attempt: 0
                  path: /uploads/movie.mkv
                  size: 734003200
                  chunk_size: 10485760
                  total_chunks: 70
                  received: [[0, 8], [10, 11]]
                  received_bytes: 115343360
                  frontier: 9
                  storage_progress: 12.86
        "400":
          x-apifox-name: Bad Query
          description: "Invalid parameter combination: pass upload_id alone, or path and size together"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Envelope"}
              example:
                code: 400
                message: "status lookup requires upload_id, or path and size"
                data: null
        "401":
          x-apifox-name: Unauthorized
          description: "Not logged in or the token has expired"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Envelope"}
              example:
                code: 401
                message: token is expired
                data: null
        "403":
          x-apifox-name: Forbidden
          description: "Path outside your allowed root, or the session belongs to another user"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Envelope"}
              example:
                code: 403
                message: multipart upload session belongs to another user
                data: null
        "404":
          x-apifox-name: Session Not Found
          description: "No live session matches the query"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Envelope"}
              example:
                code: 404
                message: multipart upload session not found
                data: null
  /api/fs/multipart/abort:
    post:
      summary: Abort multipart upload
      description: >-
        Cancel the pipeline (the driver handles it as a request cancellation —
        the local driver, for example, removes the unfinished destination
        file), discard all buffered chunks and remove the session.
      parameters:
        - {name: X-Upload-Id, in: header, required: true, schema: {type: string}, description: "session id returned by init"}
      responses:
        "200":
          x-apifox-name: Success
          description: "Session cancelled and reclaimed"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Envelope"}
              example:
                code: 200
                message: success
                data: null
        "401":
          x-apifox-name: Unauthorized
          description: "Not logged in or the token has expired"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Envelope"}
              example:
                code: 401
                message: token is expired
                data: null
        "403":
          x-apifox-name: Not Owner
          description: "The session belongs to another user"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Envelope"}
              example:
                code: 403
                message: multipart upload session belongs to another user
                data: null
        "404":
          x-apifox-name: Session Not Found
          description: "Session not found: expired, reclaimed after completion, or the server restarted"
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Envelope"}
              example:
                code: 404
                message: multipart upload session not found
                data: null
components:
  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
      bearerFormat: JWT
      description: "OpenList login token in the Authorization header"
  schemas:
    Envelope:
      type: object
      required: [code, message]
      properties:
        code: {type: integer, description: "business code (200 = success)"}
        message: {type: string, description: "success, or the error message"}
        data: {type: object, nullable: true}
    SessionSnapshot:
      type: object
      properties:
        upload_id: {type: string, description: "session id (UUID)"}
        state:
          type: string
          enum: [receiving, completed, failed_retriable, failed_permanent, aborted]
        attempt: {type: integer, description: "refill attempt counter, starts at 0"}
        path: {type: string, description: "destination path"}
        size: {type: integer, format: int64, description: "file size in bytes"}
        chunk_size: {type: integer, format: int64, description: "effective chunk size; the client must slice with this value"}
        total_chunks: {type: integer}
        received:
          type: array
          description: "inclusive index ranges already received (skip them on resume)"
          items:
            type: array
            items: {type: integer}
            minItems: 2
            maxItems: 2
        received_bytes: {type: integer, format: int64, description: "bytes received, including data already consumed by the driver"}
        frontier: {type: integer, description: "next chunk index the driver will consume"}
        storage_progress: {type: number, description: "server-to-storage upload progress, 0-100"}
        error: {type: string, description: "failure reason; present only in failed states"}
    SnapshotResp:
      allOf:
        - $ref: "#/components/schemas/Envelope"
        - type: object
          properties:
            data: {$ref: "#/components/schemas/SessionSnapshot"}

所有端点挂载于 /api/fs/multipart,需要登录(Authorization header 携带 token)。响应统一为 JSON envelope {"code": int, "message": string, "data": object}HTTP 状态码恒为 200——下文各端点状态码表(以及 OpenAPI 的 Responses)中的状态码均指 envelope 的 code 字段。

端点总览

方法 路径 中间件 作用
POST /api/fs/multipart/init Auth + FsUp(与 /fs/put 相同的写权限校验) 创建或恢复会话,并立即启动驱动上传流水线
PUT /api/fs/multipart/chunk Auth + 上传限速器 上传一个分片(幂等,可乱序并发)
POST /api/fs/multipart/complete Auth 等待流水线终态(join 点)
GET /api/fs/multipart/status Auth 查询会话(按 id,或按 path+size 做续传发现)
POST /api/fs/multipart/abort Auth 取消会话并立即回收缓冲数据

除 init 外,所有端点校验会话归属(创建者 user id),非本人返回 403;所有端点均可能返回 401(未登录 / token 失效)。

POST /init

Header 类型 必填 说明
File-Path string 目标完整路径,URL-encoded(与 /fs/put 一致)
X-File-Size int64 文件字节数,必须 > 0(空文件走 /fs/put
X-Chunk-Size int64 建议分片字节数;服务端卡到 [1MB, 管理员设置值]——客户端只能调小不能越过管理员上限,最终值以响应的 chunk_size 为准
Overwrite boolean 默认 true;为 false 时若目标已存在则拒绝
X-File-Md5 / X-File-Sha1 / X-File-Sha256 string 文件哈希;提供后:1. 随流水线先行到达驱动,支持秒传(命中则无需再传任何分片);2. 作为断点续传的文件身份凭证
Last-Modified int64 毫秒时间戳
Content-Type string 文件 MIME,缺省按扩展名推断

响应 data:SessionSnapshot(见文末)外加 resumed: bool

会话复用规则(同一用户 + 同一path + 同一size):

  • 存活 receiving 会话且哈希与原会话一致 → 复用(resumed=truereceived 给出可跳过的分片区间)。无哈希不复用——路径+大小不足以证明是同一文件,防止同尺寸不同内容的静默混合;
  • failed_retriable 会话 → 无条件复用(旧数据已全部丢弃,重灌有逐片 CRC 校验兜底);
  • 其余情况 → 旧会话被取代并清理,创建新会话。
code 说明
200 成功,data = SessionSnapshot + resumed
400 File-Path 解码失败 / X-File-Size 缺失、非法或 ≤ 0 / X-Chunk-Size 非整数
401 未登录或 token 失效
403 功能未开启 / 无写权限 / 路径越权 / Overwrite: false 且目标已存在
405 目标存储不支持上传
500 目标存储解析失败

PUT /chunk

Header 类型 必填 说明
X-Upload-Id string init 返回的会话 id
X-Chunk-Index int 0 起分片序号

请求体(application/octet-stream)= 该分片的原始字节;除最后一片外长度必须恰为 chunk_size,最后一片为余量。

行为要点:

  • 幂等:已接收或已被消费的分片重发直接成功;会话已完成(如秒传命中)时分片被幂等吸收,返回 state=completed 的快照——客户端应停止发送剩余分片;
  • 窗口:仅接收 [frontier, frontier+8) 内的分片,窗口随驱动消费向前滑动;
  • 所有响应路径都会先读完请求体再回包(避免浏览器把提前响应视为网络错误)。
code 说明
200 成功,data = 最新 SessionSnapshot(含幂等吸收的情况)
400 X-Chunk-Index 非法 / 分片长度不符 / 重灌分片与上次尝试 CRC 不一致 / 会话已处于失败态(data 携带 stateerror
401 未登录或 token 失效
403 会话属于其他用户
404 会话不存在(已过期 / 完成后已回收 / 服务重启)
409 同一分片正被另一连接上传,稍候重试;data 携带最新快照
429 窗口满:请求已在服务端停靠至多 10 秒(远低于常见 CDN 请求时限)仍无空槽。流控信号而非错误,稍候重发即可;data 携带最新快照

POST /complete

Header 类型 必填 说明
X-Upload-Id string init 返回的会话 id

阻塞等待流水线终态(与 /fs/put 等待驱动完成的语义一致),成功后会话随即回收。若连接被 CDN 提前掐断,客户端可轮询 /status 至终态。

code 说明
200 上传完成,data = state=completed 的快照
400 分片未收齐(立即返回,防止长时间挂起连接)/ 流水线失败(data 携带 stateerror
401 未登录或 token 失效
403 会话属于其他用户
404 会话不存在

GET /status

两种合法参数组合:upload_id 单独使用(组合1);或 pathsize 同时提供(组合2,续传发现:按当前用户+路径+大小查存活会话)。

Query 类型 必填 说明
upload_id string 会话 id
path string 目标完整路径,URL-encoded
size int64 文件字节数
code 说明
200 命中,data = SessionSnapshot
400 参数组合不合法(无 upload_idpath+size 不完整)
401 未登录或 token 失效
403 路径越权 / 按 id 查询到他人会话
404 无匹配会话

POST /abort

Header 类型 必填 说明
X-Upload-Id string init 返回的会话 id

取消流水线(驱动侧按请求取消处理,例如 local 驱动会删除未完成的目标文件)、删除缓冲分片、移除会话。

code 说明
200 已取消并回收
401 未登录或 token 失效
403 会话属于其他用户
404 会话不存在

SessionSnapshot 结构

字段 类型 说明
upload_id string 会话 id(UUID)
state string receiving / completed / failed_retriable / failed_permanent / aborted
attempt int 重灌次数(≥ 0)
path string 目标完整路径
size int64 文件字节数
chunk_size int64 服务端最终采用的分片字节数(客户端必须以此切片)
total_chunks int 分片总数
received [[int,int]] 已接收分片的闭区间列表(续传时可跳过)
received_bytes int64 已接收字节数(含已被驱动消费部分)
frontier int 驱动消费位点(下一个待消费分片序号)
storage_progress float 服务端→存储的真实上传进度 0–100(backending 阶段展示用)
error string 失败原因(仅失败态出现)

失败重试协议

会话 failed_retriable 时:旧分片数据已丢弃(received 为空),元数据保留 30 分钟。客户端先单独发送 chunk 0(它触发流水线重启,并发发送会与重启竞争被拒),确认后再并发发送其余分片。服务端对每片记录 CRC32,重灌分片与上一次尝试不一致时判定"文件已变化",会话转 failed_permanent

生命周期

会话仅存内存(服务重启后失效,暂存文件启动时自动清扫);30 分钟滑动无活动过期;completed 在 complete 成功后立即回收,失败态保留至 TTL 供查询错误原因。

@thetapilla thetapilla requested a review from xrgzs July 12, 2026 08:26
@thetapilla

Copy link
Copy Markdown
Author

在 Apifox 起了个空项目导入上面的 OpenAPI ,效果在这里
改改就能用吧,应该。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants