Repository navigation
[1/7] feat(core-store): local store for run logs - #1805
Merged
Merged
Conversation
Implement run-*.swanlab persistence compatible with core_python.store: 7-byte header (:SWL + 0xE1D6 + version 2), 32 KiB blocks, 7-byte chunk headers with type-seeded CRC32, zero padding at block tails. - Writer: exclusive create, chunk splitting (FULL/FIRST/MIDDLE/LAST), record offsets including block padding - Reader: scan from a byte offset with an optional upper bound, report io.EOF / ErrTruncated / ErrCorrupt / ErrBadHeader distinctly - AsyncWriter: quantitative, timed and close flush paths; publishes (flushedStoreSeq, flushedOffset) only after a full record is written and flushed; OnError fires once so the session can set storeDegraded Does not touch proto, manager or the storeSeq->offset mapping (PR-11).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概要
新增
run-*.swanlab本地持久化:一条 Record 可靠落盘、原样读回,进程崩溃留下的半条记录不会污染读取。文件格式与 Pythoncore_python.store完全一致。变更内容
只新增
core/internal/store,不改动 proto / manager / service,零新增第三方依赖。store.goWriter(Create排他创建、FULL/FIRST/MIDDLE/LAST 分块、块尾补零、偏移记账、Flush与幂等Close);Reader(OpenOffset(path, offset, limit)从任意记录边界起读并以limit为上界,Scan区分 EOF / 截断 / 损坏 / 坏文件头,Position供增量扫描)async.goAsyncWriter:定量 / 定时 / Close 三条刷盘路径,原子发布(flushedStoreSeq, flushedOffset),OnError供上层置storeDegraded关键行为:
Flush成功后才发布,因此并发 Reader 以flushedOffset为上界时不会读到半条跨块记录;发布允许滞后,但不会超前。OnError,之后丢弃本地写入但继续消费入口,不阻断上传主链路。Reader.Scan遇到尾部半条记录返回ErrTruncated,此前读出的记录仍然有效;CRC 或帧结构非法返回ErrCorrupt。Writer.Offset()在异步写入下只反映已被消费的部分,异步场景应使用Flushed()或事后用Reader扫描。验证
go build ./...、go vet ./...、go test -race ./...通过;internal/store语句覆盖率 90.0%;golangci-lint(仓库.golangci.yml)0 issues。flushedOffset、AsyncWriter 三条刷盘路径与降级丢弃。core_python.store双向互读 200 条混合记录:条数、num序列、每条 payload sha256 一致;两语言对同一记录序列写出的文件整份字节相同;截断与定点读取符合预期。Base:
feat/core/io-model。