Repository navigation
Feat/core/io model - #1804
Draft
Nexisato wants to merge 6 commits into
Draft
Feat/core/io model#1804Nexisato wants to merge 6 commits into
Nexisato wants to merge 6 commits into
Conversation
统一记录标签、固定槽位号与字节计量口径，供 store / builder / intake / uploader 并行开发时共同依赖，避免各自定义平行结构。
This was referenced Oct 28, 2026
* feat(core): add local store for run logs Implement run-*.swanlab persistence compatible with core_python.store: 7-byte header (:SWL + 0xE1D6 + version 2), 32 KiB blocks, 7-byte chunk headers with type-seeded CRC32, zero padding at block tails. - Writer: exclusive create, chunk splitting (FULL/FIRST/MIDDLE/LAST), record offsets including block padding - Reader: scan from a byte offset with an optional upper bound, report io.EOF / ErrTruncated / ErrCorrupt / ErrBadHeader distinctly - AsyncWriter: quantitative, timed and close flush paths; publishes (flushedStoreSeq, flushedOffset) only after a full record is written and flushed; OnError fires once so the session can set storeDegraded Does not touch proto, manager or the storeSeq->offset mapping (PR-11). * fix(core-store): pass golangci-lint and align header errors * refactor(core-store): fold header, reader and writer into store.go * chore: simplify comments
Nexisato
added
💪 enhancement
* chore(core): ignore smoke staging directory
Keep `core/.smoke-tmp/` out of git: the playground smoke runner copies
`smoke/go/sN` there and runs it inside the module so it can import
`core/internal/**` without shipping temporary packages.
* feat(core-proto): pass cloud credentials to core via CoreSettings
Add api_key (11), api_host (12) and web_host (13) so the Go core can
authenticate against the cloud with the same field numbers as the
ApiKey auth refactor, and regenerate both the Go and Python stubs.
api_key is a secret: it is only transported over DeliverRunStart, never
via command line arguments, and must be masked in any log output.
* feat(core-client): own the cloud transport layer
Keep business endpoints out of this package: it exposes a single
`Do(Request) (*Response, error)` round trip and owns everything about
how a request is sent.
- shared *http.Transport pool, per (api_host, sha256(api_key)) Client
reuse, 60 s timeout, 5 retries, 0.5 backoff on 429/5xx and the
X-SwanLab-SDK-Version header
- auth: stateless `Authorization: ApiKey` with a one-shot
/api/auth/verify probe; legacy deployments fall back to sid cookies
with a 7-day refresh window and locked relogin on 401
- APIError{Retryable, Code, Message, TraceID} with IsRetryable /
IsAuthenticationError / IsInvalidResponse / IsHouseRejected, plus
Response.HouseRejected for "202 instead of 200" writes
- clienttest: shared httptest fake cloud and injectable clock, reused by
the client and api test suites
* feat(core-api): wrap the swanlab rest endpoints
Layer the REST protocol on top of internal/client so transport policy
and protocol semantics can evolve separately: `api` depends on `client`
and never the other way around.
- model.go: every request/response body and protocol enum in one place
(Experiment, Project, Summary, ColumnUpload, PrepareSaveFile,
MultipartFile, ResumeMode, RunState, MetricKind, ...), no more
endpoint-local anonymous structs
- api.go: project / experiment / metrics / series / presigned / save /
profile / heartbeat
- result rules: House accepts only 200 (202 is a non-retryable
rejection), experiment create distinguishes 200/201, createdAt and
finishedAt are taken at request time, prepare responses are checked
for element count and empty entries
* fix(core-client): tune the shared pool, unblock the factory lock and deflake the cancel test
- raise MaxIdleConnsPerHost from 2 to 16 (64 overall) on the cloned
transport: the upload pipeline drives several tracks and file uploads
concurrently, and the default value keeps dropping keep-alives
- move Factory.Client's auth probe outside the mutex so one slow identity
cannot block other session startups; concurrent callers for the same
identity still converge on the first cached instance
- drive TestContextCancellationIsNotRetried from a handler signal instead
of a 20 ms sleep, which races with request dispatch on loaded CI runners
- cover both changes with pool-parameter and concurrent-factory tests
* docs(core-api): record the empty-batch contract and the stricter 202 rule
Both notes exist to stop a future reader from "fixing" the current
behaviour by accident:
- empty batches are the sender's responsibility: Python short-circuits in
its api layer, but Go takes `metrics any`, so empty-checking here would
need reflection; the pipeline knows the concrete slice
- House 202 is rejected on purpose even though Python's `response.ok`
accepts it, so silently dropped writes surface as errors
* chore: tighten connection pool
* chore: refine comments
* fix(core-client): align 401 handling with python and release idle connections
- forward CloseIdleConnections through retryTransport: Factory.Close was a
no-op because the wrapper never exposed the method, so idle connections
were never released
- drop the 401 relogin and replay path: Python treats 401 as a plain auth
error, and the loop could trigger one login per concurrent stale request
- classify auth response read failures as transport errors instead of
protocol errors, so a truncated body keeps its retry semantics
- normalize api_host via parsed.Host so IPv6 literals keep their brackets
* fix(core-api): reject malformed prepare responses
Validate presigned and Save prepare responses up front instead of letting
empty URLs, a missing uploadId or a broken part list reach the upload stage,
where they would surface as retryable network errors:
- urls must match the request size and every entry must be non-empty
- uploadId must be non-empty
- part count must match the requested count, part numbers must be 1..N with
a URL each
* chore: update golangci lint
) * feat(core-runtime): add run settings, context and file name helpers Add the runtime foundation for a single run: config.Settings mirrors CoreSettings as a pure-Go type with Python-aligned defaults and validation, runctx.RunContext derives local directories and carries online parameters, and pkg/fs ports the file-name safe_fmt / safe_truncate helpers. pkg/fs behaviour is locked by Python-generated fixtures so the Go and Python implementations stay byte-compatible. * feat(core-runtime): add metrics state machine and record builder Add RunMetrics with separate read-only precheck and commit phases, roaring64 step deduplication, finite-float extremes and resume-summary restoration. Add the stateless record builders (fixed slots, positive numbering, auto and resume columns, default name/colour vocabulary) and the concurrent counter. This pulls in github.com/RoaringBitmap/roaring/v2. * fix(core-runtime): keep run file test portable on windows The test hard-coded "/rd" as the run directory, so filepath.Dir returned a backslash path on Windows. Build the expected directory with filepath.Join instead. * fix(core): step range constraint * chore: sync bench with py version bitmap
…1809) * [4/7] feat(core-intake): quota, admission and dual-channel routing - add internal/budget: session-level record/byte quota with acquire, release and estimate adjustment - add internal/buffer: two-phase intake, storeSeq assignment, pause and degraded backpressure, routing to store/upload - cover quota conservation, backpressure, routing and Close paths * [4/7] fix(core-intake): harden pause recovery and upload backpressure - 暂停记录保留到 Flushed 确认，写盘失败由 DrainRecovery 补传 - Resume 后的旧票据开启新的回填区间 - 暂停后降级仍受上传预算约束 - 上传通道槽位纳入预留，Commit 不再持锁等待 - 归并容量等待与回收发送逻辑，恢复逻辑拆到 recovery.go * [4/7] fix(core-intake): preserve upload order and deferred quota - 降级批次并入回收队列按序补传，不再越过更早的暂停记录 - Demand 拆分 Deferred：END 保存从 Reserve 起占用配额，预算不足直接反压 - 全局暂停状态推迟到 Commit 发布，放弃票据不再遗留 paused - Close 补投回收记录被取消时一并归还 pending * [4/7] fix(core-intake): charge deferred saves by their own bytes - Demand 增加 DeferredBytes，Bytes 仅覆盖立即上传记录 - 暂停批次只按延迟记录自身的条数与字节申请配额 - 补充大立即记录叠加小 END 保存的回归测试 * chore: simplify code * chore: update gitignore
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description