Repository navigation
tests: driver/postgres TestStreamingLarge measures the streaming path, and the Query_1col_1row budget is per path - #1144
Merged
Conversation
…ve heap while it does (Refs #1104) With the sync fast path a poll that takes the whole reply buffers every row on purpose, and whether it does is timing: on -race -cover runs it did in 6 of 8 iterations, and 11.4 MB of buffered rows (held by the request pool past the test's one GC) failed the 10 MB streaming budget. The conn runs without the fast path (a platform without one has none), so the request streams, and the live heap is sampled after two GCs every 5000 rows and at the end.
… elsewhere (Refs #1104) Without the sync loop simpleQuery allocates the colsCh before it enqueues: 15 allocs/op, steady (linux with syncLoop nil: 20 of 20; darwin native: 3 of 3; the Windows job logged 15), against 14 in 50 of 50 runs on linux, 30 of 30 under six CPU hogs. A per-path budget, no headroom.
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…he allocation budget follows the conn's path (Refs #1104) The comment said the sync path's buffering is capped at maxDirectResultBytes. It is checked after the append and cannot stop the poll, so it is not a memory bound above the cap; say so, and that the test exercises the streaming path only. The Query_1col_1row budget is keyed on whether the bench's conn has a syncLoop, the condition that decides the extra allocation, and its comment is shorter.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Flakes lane, package D (+ F2): driver/postgres
TestStreamingLargemeasured the wrong path and the wrong moment; and theQuery_1col_1rowallocation budget is a per-path number. Two test commits, no driver change. Refs #1104D.
TestStreamingLarge(stream_test.go)Failure: "heap grew 11.3 MB, exceeds 10 MB budget — likely not streaming" (CI, a dependabot PR, green on rerun). I reproduced it: the package with
-race -cover -count=8fails 6 of 8 iterations, always with 11,9xx,xxx bytes (11.4 MB).Cause, from a heap profile diff (before/after the query) of a failing iteration: the whole 11.35 MB is
pgConn.QueryContext->simpleQuery->pollSync->pgSyncLoop.WriteAndPoll->onRecv->appendBuffered->appendRowFromAlias. That is the SYNC fast path: when the poll on the caller's goroutine takes the whole reply,syncModeis on and every row is buffered, and the request never streams. Whether the poll takes all 100K rows in one go is timing, so on a slow instrumented run it does (the buffered result, 100K x about 119 bytes, is then held by the request pool, and the test's singleruntime.GC()leaves it insync.Pool's victim cache) and on others the poll gives up early and the request streams. At 100K rows (under the 64 MiBmaxDirectResultBytes) that buffering is within what the driver promises; the test assumed one path and measured retained heap after the rows were closed.Fix: the test runs the conn without its sync fast path (
conn.syncLoop = nil, which is what a platform without one has), so the request streams on every run, which is the property under test; it samples the LIVE heap (two GCs, so pool victims go too) every 5000 rows while the rows are consumed and once at the end, and the 10 MB budget applies to the largest sample. That also makes the assertion stronger: a result buffered whole shows up mid-stream, not only if something retains it afterwards.-race -cover -count=8-race -cover -count=6, onceok(384 s); growth 0.06 to 0.10 MB each-race -count=5promoteToStreamingdisabled (if false && ...)What this test does not cover, stated exactly. It exercises the streaming path (
syncLoop = nil); the sync fast path's whole-reply buffering and the fall-through from it to streaming with rows already buffered are not covered. The sync path's memory is NOT bounded bymaxDirectResultBytes:appendBufferedchecks that cap after it appends the row and cannot stop the poll, so above the cap a reply is still buffered whole beforeErrResultTooBig. The reviewer measured a 2M-row reply (about 240 MB on the wire) at 1.1 to 1.2 GB peak heap when the client is slower than the sender (-raceand-covertogether, which is also what made the 100K-row test flake). That is a separate, pre-existing defect in the driver, not touched here and not hidden by this change (the old 10 MB budget at 100K rows was a streaming expectation the sync path never promised under 64 MiB); it is handled in the separate PR on the sync-path buffering bound.F2.
TestAllocBudgets/Query_1col_1row(alloc_guard_test.go)The Windows portable-fallback job of #1143 (run 40521342083) logged
allocs/op=15 (budget 14). In that log the subtest PASSED: the Portable fallback step does not setTESTING_STRICT_ALLOC_BUDGETS, so the line is informational, and the job failed for another reason (see below). But the 15 is real, and it is not noise:--cpus 2,-count=50-count=30syncLoop = nil(the path of a platform without the sync poll)-count=3The +1 is the
colsChthatsimpleQueryallocates before it enqueues when there is no sync loop (if !c.useDirect && c.syncLoop == nil).AllocsPerOpis an integer division (floor), so one stray background allocation cannot turn a 14 into a 15; the 15 is a steady state. The budget was measured on linux only, with zero headroom, which is right for the path it measured.Fix: the budget follows the path the bench's conn took (
conn.syncLoop != nil: 14, else 15), not the OS, with the evidence in a comment. No headroom added: linux stays at 14 (strict, exact), and the other platforms' number is the one measured. With it, the test passes strict on this Mac.What the Windows job actually failed on
Not
Query_1col_1row. The postgres package's output in that job ends at=== RUN TestTxDoneAfterFailedCommitfollowed byFAIL github.com/goceleris/celeris/driver/postgres 6.158s, with no panic, noexit statusline and no--- FAIL: the test binary ended there without a message (the test deliberately causes a recovered nil-interface panic inTx.Rollback, and exists since v1.4.0). I cannot reproduce on Windows. It looks like a runner or process-level event, not an assertion; a rerun is the only evidence available. I did not change anything for it.Whole package and lint
golang:1.27.3,--cpus 2,-race -cover -count=6whole./driver/postgres: ok (above). Strict allocation guards, linux,-count=3: 14 (budget 14) and 0 (budget 0).golangci-lint(CI pin v2.14, linux): 0 issues;go veton linux, darwin and windows. (A local windows-targeted run shows two findings,rowch_failall_887_test.go:311errcheck and anolintlintinconn_netconn.go, both on main and not in this diff.)Release note
None (tests only).