Repository navigation
[4/7] feat(core-intake): quota, admission and dual-channel routing - #1809
Merged
Merged
Conversation
- add internal/budget: session-level record/byte quota with acquire, release and estimate adjustment - add internal/buffer: two-phase intake, storeSeq assignment, pause and degraded backpressure, routing to store/upload - cover quota conservation, backpressure, routing and Close paths
Nexisato
marked this pull request as ready for review
October 29, 2026 05:29
- 暂停记录保留到 Flushed 确认,写盘失败由 DrainRecovery 补传 - Resume 后的旧票据开启新的回填区间 - 暂停后降级仍受上传预算约束 - 上传通道槽位纳入预留,Commit 不再持锁等待 - 归并容量等待与回收发送逻辑,恢复逻辑拆到 recovery.go
- 降级批次并入回收队列按序补传,不再越过更早的暂停记录 - Demand 拆分 Deferred:END 保存从 Reserve 起占用配额,预算不足直接反压 - 全局暂停状态推迟到 Commit 发布,放弃票据不再遗留 paused - Close 补投回收记录被取消时一并归还 pending
- Demand 增加 DeferredBytes,Bytes 仅覆盖立即上传记录 - 暂停批次只按延迟记录自身的条数与字节申请配额 - 补充大立即记录叠加小 END 保存的回归测试
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概要
新增会话级的接收与分流层:一条 record 进来先预留容量,再分配
storeSeq并同时进入本地落盘与上传两条通道。本地run-*.swanlab是唯一事实源;容量不足只反压,不丢记录;上传暂停期间尚未确认刷盘的记录在本地写失败时会按序回收补传。变更内容
internal/budgetbudget.goTryAcquire/Acquire/Release/Adjust与水位统计internal/bufferbuffer.goReserve/Commit;storeSeq分配;本地/上传双通道路由;容量预留与反压internal/bufferrecovery.gopauseStoreSeq与本地失效后的按序回收补传关键行为:
Reserve只读预检并预留本地通道、上传预算与上传槽位,失败零副作用;Commit分配storeSeq、写本地并分流。Demand分别声明立即/延迟记录的条数与字节估算,暂停批次只按延迟记录自身占用配额。QueueWait内有界等待,仍失败返回ErrOverloaded;单批超过通道容量返回ErrBatchTooLarge。只有 AsyncWriter 真实写入失败才置storeDegraded,运行期不会修改skip_store等启动配置。pauseStoreSeq在记录真正提交时才发布;被放弃、空提交或校验失败的票据只归还容量,不会遗留暂停。恢复后旧票据若晚到,会开启新的回填区间。SAVE_POLICY_END记录从Reserve起按自身条数与字节占用上传预算,直到Close上传终结;暂停期间预算不足直接反压,不会以未计费状态在内存中无界堆积,也不会被同批立即记录的字节拖累。Flushed()推进;写盘失败后进入统一回收队列,后续批次排在队尾按 FIFO 补传,不会越过更早的记录。宁可短暂超出预算也不丢记录;降级后不再使用暂停快路径,继续受上传预算约束。Start(-1)/Finish(-2)仅本地;崩溃error_log仅上传;SAVE_POLICY_END先落本地、Close时再上传。Close先补投回收队列与暂存的 END 保存,再关闭上传通道;被取消时两个队列都会恢复,重试不会丢失记录。验证
go build ./...、go test -race ./...通过;golangci-lint run0 issues。budget100%,buffer88.9%。AsyncWriter写失败后的记录回收。storeDegraded后反压、慢盘反压三种场景通过;落盘文件经 PythonDataStoreReader回读,num序列连续无洞。范围说明
budget+intake;攒批、分轨、HTTP 上传在后续 PR。internal/config仅补充SkipStore的启动期/运行期边界注释。Base:
feat/core/io-model.