Repository navigation
[3/7] feat(core-runtime): run context, metrics and record builder - #1807
Merged
Merged
Conversation
Nexisato
marked this pull request as ready for review
October 28, 2026 09:07
Add the runtime foundation for a single run: config.Settings mirrors CoreSettings as a pure-Go type with Python-aligned defaults and validation, runctx.RunContext derives local directories and carries online parameters, and pkg/fs ports the file-name safe_fmt / safe_truncate helpers. pkg/fs behaviour is locked by Python-generated fixtures so the Go and Python implementations stay byte-compatible.
Add RunMetrics with separate read-only precheck and commit phases, roaring64 step deduplication, finite-float extremes and resume-summary restoration. Add the stateless record builders (fixed slots, positive numbering, auto and resume columns, default name/colour vocabulary) and the concurrent counter. This pulls in github.com/RoaringBitmap/roaring/v2.
The test hard-coded "/rd" as the run directory, so filepath.Dir returned a backslash path on Windows. Build the expected directory with filepath.Join instead.
Nexisato
force-pushed
the
feat/core/io-3-runtime
branch
from
October 28, 2026 09:28
d35a221 to
c02880d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概要
新增会话运行时:一个 run 知道自己是谁(配置/路径)、收过哪些指标(位图与极值)、每条记录该编几号。为后续 intake / uploader / manager 提供纯 Go 的配置、上下文与记录组装。
变更内容
internal/configsettings.goSettings与CoreSettings一一对应(不 import proto);Defaults()/Validate()对齐 Python 默认值与边界;MaskKey/String()日志脱敏internal/runctxrunctx.goRunContext内嵌配置,派生media/、files/、run-{run_id≤100}.swanlab路径,承载OnlineParamsinternal/pkg/fsfs.goSafeFmt/SafeTruncate:文件名安全化与 UTF-8 截断internal/metricsmetrics.go/resume.goRunMetrics:两阶段预检/提交、roaring64step 去重、有限浮点极值、Resume 摘要恢复internal/pkg/builderbuilder.go/column.go/naming.gointernal/pkg/countercounter.go关键行为:
Precheck*(只读)与Commit*(写入),供上层先预留容量再改状态;预检失败不触碰 step 位图。Commit报错时仍须投递列记录,与 Pythonsafe.block的产出顺序一致。roaring64,每 4096 个新 step 执行一次RunOptimize;step ≤ min_step、重复 step、step < 0、step > 2^48分别处理(负 step 对齐 PythonOverflowError)。min_step水位线拦截历史 step;UNSPECIFIED 媒体占位列放行任意媒体类型。config.Validate与 Python pydantic 边界一致:record_batch ∈ (0, 100000)、record_interval > 0、save_part > 0;其余 save_* 不额外约束。runctx只计算路径、不创建目录;文件名工具pkg/fs的行为由 Python 生成的 fixture 锁定,跨语言差分 2490 个用例无差异。依赖:新增
github.com/RoaringBitmap/roaring/v2。CoreSettings的 11/12/13 字段由 #1806 提供,本 PR 只消费纯 Go 的config.Settings。性能对比
基准与
tests/benchmark/sdk/internal/core_python/metrics/bench_metrics_steps.py同规模、同场景:200 key × 5000 step(100 万 step),-benchtime=1x -count=3取中位数(Apple M4 Pro,Go 1.27.3 / CPython 3.11.15)。AcceptbitmapSteps(生产路径)Accept只测RunMetrics.accept,与 Pythontry_accept_step同口径;Steps走CommitScalar完整生产路径(含列校验、记录组装)。set基线在 sparse 更快(210.8 ns/step),但常驻内存高 360×(sequential 662 KB/key vs bitmap 1.84 KB/key)。复现:
验证
go build ./...、go vet ./...、go test -race ./...通过;golangci-lint run0 issues。core_python差分 25 条记录(含重复 step、min_step=-1、超限、类型冲突、Resume 水位线 88/100),经store落盘后由 PythonDataStoreReader回读,条数 /num序列 / 列内容 / 目录结构一致;44 个名称 / 颜色 slug 对齐。Base:
feat/core/io-model.