Skip to content

[1/7] feat(core-store): local store for run logs - #1805

Merged
Nexisato merged 4 commits into
feat/core/io-modelfrom
feat/core/io-1-store
Oct 28, 2026
Merged

Nexisato merged 4 commits into
feat/core/io-modelfrom
feat/core/io-1-store

Conversation

@Nexisato

@Nexisato Nexisato commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

概要

新增 run-*.swanlab 本地持久化:一条 Record 可靠落盘、原样读回,进程崩溃留下的半条记录不会污染读取。文件格式与 Python core_python.store 完全一致。

变更内容

只新增 core/internal/store,不改动 proto / manager / service,零新增第三方依赖。

文件 内容
store.go 文件格式(7 字节文件头、32 KiB 分块、chunk 类型、带类型基值的 CRC32、错误哨兵);Writer(Create 排他创建、FULL/FIRST/MIDDLE/LAST 分块、块尾补零、偏移记账、Flush 与幂等 Close);Reader(OpenOffset(path, offset, limit) 从任意记录边界起读并以 limit 为上界,Scan 区分 EOF / 截断 / 损坏 / 坏文件头,Position 供增量扫描)
async.go AsyncWriter:定量 / 定时 / Close 三条刷盘路径,原子发布 (flushedStoreSeq, flushedOffset),OnError 供上层置 storeDegraded

关键行为:

  • 进度只在整条记录的全部物理分块写入且显式 Flush 成功后才发布,因此并发 Reader 以 flushedOffset 为上界时不会读到半条跨块记录;发布允许滞后,但不会超前。
  • 首次写入或刷盘失败回调一次 OnError,之后丢弃本地写入但继续消费入口,不阻断上传主链路。
  • Reader.Scan 遇到尾部半条记录返回 ErrTruncated,此前读出的记录仍然有效;CRC 或帧结构非法返回 ErrCorrupt。
  • Writer.Offset() 在异步写入下只反映已被消费的部分,异步场景应使用 Flushed() 或事后用 Reader 扫描。

验证

  • go build ./...、go vet ./...、go test -race ./... 通过;internal/store 语句覆盖率 90.0%;golangci-lint(仓库 .golangci.yml)0 issues。
  • 单测覆盖跨两块 / 三块、块尾补零、空记录、字节布局、排他创建、CRC 损坏、非法 chunk 类型、非零填充、半条记录、偏移与 limit、并发读不越过 flushedOffset、AsyncWriter 三条刷盘路径与降级丢弃。
  • 与 Python core_python.store 双向互读 200 条混合记录:条数、num 序列、每条 payload sha256 一致;两语言对同一记录序列写出的文件整份字节相同;截断与定点读取符合预期。

Base: feat/core/io-model。

Implement run-*.swanlab persistence compatible with core_python.store:
7-byte header (:SWL + 0xE1D6 + version 2), 32 KiB blocks, 7-byte chunk
headers with type-seeded CRC32, zero padding at block tails.

- Writer: exclusive create, chunk splitting (FULL/FIRST/MIDDLE/LAST),
  record offsets including block padding
- Reader: scan from a byte offset with an optional upper bound, report
  io.EOF / ErrTruncated / ErrCorrupt / ErrBadHeader distinctly
- AsyncWriter: quantitative, timed and close flush paths; publishes
  (flushedStoreSeq, flushedOffset) only after a full record is written
  and flushed; OnError fires once so the session can set storeDegraded

Does not touch proto, manager or the storeSeq->offset mapping (PR-11).
@Nexisato Nexisato changed the title [1/7] feat(core): local store for run logs [1/7] feat(core-store): local store for run logs Oct 28, 2026
@Nexisato Nexisato changed the title [1/7] feat(core-store): local store for run logs [1/7] feat(core-store) Oct 28, 2026
@Nexisato Nexisato changed the title [1/7] feat(core-store) [1/7] feat(core-store): local store for run logs Oct 28, 2026
@Nexisato
Nexisato marked this pull request as ready for review October 28, 2026 07:23
@Nexisato
Nexisato merged commit 539adb7 into feat/core/io-model Oct 28, 2026
8 checks passed
@Nexisato
Nexisato deleted the feat/core/io-1-store branch October 28, 2026 07:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant