Skip to content

[3/7] feat(core-runtime): run context, metrics and record builder - #1807

Merged
Nexisato merged 5 commits into
feat/core/io-modelfrom
feat/core/io-3-runtime
Oct 29, 2026
Merged

Nexisato merged 5 commits into
feat/core/io-modelfrom
feat/core/io-3-runtime

Conversation

@Nexisato

@Nexisato Nexisato commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

概要

新增会话运行时:一个 run 知道自己是谁(配置/路径)、收过哪些指标(位图与极值)、每条记录该编几号。为后续 intake / uploader / manager 提供纯 Go 的配置、上下文与记录组装。

变更内容

包 文件 职责
internal/config settings.go Settings 与 CoreSettings 一一对应(不 import proto);Defaults() / Validate() 对齐 Python 默认值与边界;MaskKey / String() 日志脱敏
internal/runctx runctx.go RunContext 内嵌配置,派生 media/、files/、run-{run_id≤100}.swanlab 路径,承载 OnlineParams
internal/pkg/fs fs.go SafeFmt / SafeTruncate:文件名安全化与 UTF-8 截断
internal/metrics metrics.go / resume.go RunMetrics:两阶段预检/提交、roaring64 step 去重、有限浮点极值、Resume 摘要恢复
internal/pkg/builder builder.go / column.go / naming.go 固定槽位与递增编号、自动列 / Resume 列、默认实验名与颜色词表
internal/pkg/counter counter.go 并发安全自增计数器

关键行为:

  • 指标接收拆成 Precheck*(只读)与 Commit*(写入),供上层先预留容量再改状态;预检失败不触碰 step 位图。
  • 新列在 step 被拒(重复 / 过期 / 超限)时仍会定义,调用方 Commit 报错时仍须投递列记录,与 Python safe.block 的产出顺序一致。
  • step 去重使用 roaring64,每 4096 个新 step 执行一次 RunOptimize;step ≤ min_step、重复 step、step < 0、step > 2^48 分别处理(负 step 对齐 Python OverflowError)。
  • Resume 只重建列的 key / type / class,位图从空开始,靠 min_step 水位线拦截历史 step;UNSPECIFIED 媒体占位列放行任意媒体类型。
  • config.Validate 与 Python pydantic 边界一致:record_batch ∈ (0, 100000)、record_interval > 0、save_part > 0;其余 save_* 不额外约束。
  • runctx 只计算路径、不创建目录;文件名工具 pkg/fs 的行为由 Python 生成的 fixture 锁定,跨语言差分 2490 个用例无差异。

依赖:新增 github.com/RoaringBitmap/roaring/v2。CoreSettings 的 11/12/13 字段由 #1806 提供,本 PR 只消费纯 Go 的 config.Settings。

性能对比

基准与 tests/benchmark/sdk/internal/core_python/metrics/bench_metrics_steps.py 同规模、同场景:200 key × 5000 step(100 万 step),-benchtime=1x -count=3 取中位数(Apple M4 Pro,Go 1.27.3 / CPython 3.11.15)。

场景 Go Accept Python bitmap 提升 Go Steps(生产路径) 生产路径 vs Python
sequential 25.0 ns/step 103.8 ns/step 4.1× 47.8 ns/step 2.2×
backfill 23.0 ns/step 122.9 ns/step 5.3× 45.5 ns/step 2.7×
sparse 6,978 ns/step 18,513 ns/step 2.7× 6,988 ns/step 2.6×
  • Accept 只测 RunMetrics.accept,与 Python try_accept_step 同口径;Steps 走 CommitScalar 完整生产路径(含列校验、记录组装)。
  • 位图体积(serialize):sequential / backfill 均为 27 B/key,与 Python 一致;sparse 107,766 vs 107,848 B/key(差 0.08%)。
  • sparse 是病态稀疏(step 随机落在 2^48 空间),位图在两种语言下都退化;Go 仍比现行 Python 位图实现快 2.6×。
  • 旧 set 基线在 sparse 更快(210.8 ns/step),但常驻内存高 360×(sequential 662 KB/key vs bitmap 1.84 KB/key)。

复现:

cd core && go test -run '^$' -bench 'Steps|Accept' -benchtime 1x -count 3 ./internal/metrics
uv run pytest tests/benchmark/sdk/internal/core_python/metrics/bench_metrics_steps.py -s

验证

  • go build ./...、go vet ./...、go test -race ./... 通过;golangci-lint run 0 issues。
  • 覆盖率:config 100% / runctx 94.7% / fs 100% / metrics 99.2% / builder 100% / counter 100%。
  • S3 冒烟(本地临时目录,无网络):与 core_python 差分 25 条记录(含重复 step、min_step=-1、超限、类型冲突、Resume 水位线 88/100),经 store 落盘后由 Python DataStoreReader 回读,条数 / num 序列 / 列内容 / 目录结构一致;44 个名称 / 颜色 slug 对齐。
  • S1 回归:200 条记录 Python ↔ Go 双向互读一致。
  • CI(Lint + Linux/macOS/Windows 测试)全部通过。

Base: feat/core/io-model.

@Nexisato
Nexisato marked this pull request as ready for review October 28, 2026 09:07
Add the runtime foundation for a single run: config.Settings mirrors CoreSettings as a pure-Go type with Python-aligned defaults and validation, runctx.RunContext derives local directories and carries online parameters, and pkg/fs ports the file-name safe_fmt / safe_truncate helpers.

pkg/fs behaviour is locked by Python-generated fixtures so the Go and Python implementations stay byte-compatible.
Add RunMetrics with separate read-only precheck and commit phases, roaring64 step deduplication, finite-float extremes and resume-summary restoration. Add the stateless record builders (fixed slots, positive numbering, auto and resume columns, default name/colour vocabulary) and the concurrent counter.

This pulls in github.com/RoaringBitmap/roaring/v2.
The test hard-coded "/rd" as the run directory, so filepath.Dir returned a backslash path on Windows. Build the expected directory with filepath.Join instead.
@Nexisato
Nexisato force-pushed the feat/core/io-3-runtime branch from d35a221 to c02880d Compare October 28, 2026 09:28
@Nexisato
Nexisato merged commit c63fe3b into feat/core/io-model Oct 29, 2026
5 checks passed
@Nexisato
Nexisato deleted the feat/core/io-3-runtime branch October 29, 2026 03:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant