Repository navigation
Index topic workflow calls for bounded reporting - #304
Conversation
Backend unit coveragePASS · minimum combined coverage 98.00%. Tests: 10792 passed, 0 failed, 0 skipped. Source files: 367.
Unit coverage includes every backend workspace. PostgreSQL/Redis integration tests run separately. Commit: ae25dc1 · HTML, JSON, XML reports and job logs · attempt 1 |
PR performance resultsCurrent PR head: Token usage benchmarkPending: no results have been published. Reader API performanceSource head: PR reader API performanceResult: INCONCLUSIVE 7 inconclusive · 1 within threshold. Zero request failures across all six runs. Base Three independent GitHub-hosted ARM64 runners, each measuring base and PR sequentially with alternating order and isolated resources for every sample. 16 users, completed bootstrap and 15 seconds of warmup before each full 60-second measurement, fixed journey seed 294, 1,000 synthetic articles, cache off, five PgBouncer slots. Latency comparisonAll latency values are in milliseconds; lower is better. Values are medians of three runs, with one-millisecond histogram bins.
Additional latency and throughput metricsEach cell shows base → PR. Latencies are in milliseconds.
All paired p95 samples and changesEach pair shares a validated runner. Cells show base → PR p95 (absolute and percentage change), in milliseconds. Slower/faster counts use the same strict 20% and 20 ms thresholds against that pair's own baseline.
Individual runs
How to interpret this reportRegressed / Improved: a p95 regression needs >20% and >20 ms increase in the medians and in at least two same-runner base/PR pairs. Improvements use the inverse threshold. Within threshold means the change did not meet both thresholds. Inconclusive: a within-revision p95 range >35% of its median and >20 ms is too noisy for a confident latency verdict. A material median change without two confirming pairs is also inconclusive; this does not prove unchanged performance. Inconclusive latency comparisons are non-blocking: the check passes without claiming an improvement. Confirmed regressions and test/report errors fail the PR gate. Throughput is descriptive: paced users do not measure maximum capacity. This uncached public API test does not establish production health, browser performance or worker capacity. Base and PR share a runner within each pair; three pairs use independent runners. Alternating order reduces host-assignment bias but cannot eliminate changes in host load over time. Full HTML/CSV reports and logs · attempt 1 |
Topic report historyRead result: IMPROVED. Write result: HIGHER LATENCY OBSERVED. Native ARM64 PostgreSQL 17 · 90-day concurrent reports. Delta = after - before. Improvement = (before - after) / before × 100: positive means less time/work; negative means slower or higher cost. Tested commit: 100,000 articlesPostgreSQL 17.11 / ARM64 / work_mem 4MB. 6,000 workflows · 18,000 topic calls · 500,000 inference rows.
Exact output parity: 1 / 7 / 30 / 90 days passed. Observed checks: 4/4. 4 report workers; 12 report pairs total; 200 ingestion transactions per strategy. Equivalent backfill: 0.412 s. Correction full-page images: 0 → 1. The synthetic writes and correction above roll back and exclude commit latency; they cannot establish durable write throughput or overall production benefit. Write result: HIGHER LATENCY OBSERVED. Committed topic operations follow. Committed topic writes: 5 calls per workflow3 alternating repeats; 600 iterations and 2,400 committed transactions per strategy. Source histories match and normalized facts were verified.
WAL full-page images: 609 → 808. WAL is a server-wide snapshot sensitive to full-page images and background activity; it is not a stable amplification ratio. Committed topic writes: 20 calls per workflow3 alternating repeats; 600 iterations and 2,400 committed transactions per strategy. Source histories match and normalized facts were verified.
WAL full-page images: 129 → 91. WAL is a server-wide snapshot sensitive to full-page images and background activity; it is not a stable amplification ratio. 300,000 articlesPostgreSQL 17.11 / ARM64 / work_mem 4MB. 18,000 workflows · 54,000 topic calls · 1,500,000 inference rows.
Exact output parity: 1 / 7 / 30 / 90 days passed. Observed checks: 4/4. 4 report workers; 12 report pairs total; 200 ingestion transactions per strategy. Equivalent backfill: 1.295 s. Correction full-page images: 1 → 0. The synthetic writes and correction above roll back and exclude commit latency; they cannot establish durable write throughput or overall production benefit. Write result: HIGHER LATENCY OBSERVED. Committed topic operations follow. Committed topic writes: 5 calls per workflow3 alternating repeats; 600 iterations and 2,400 committed transactions per strategy. Source histories match and normalized facts were verified.
WAL full-page images: 2,021 → 1,647. WAL is a server-wide snapshot sensitive to full-page images and background activity; it is not a stable amplification ratio. Committed topic writes: 20 calls per workflow3 alternating repeats; 600 iterations and 2,400 committed transactions per strategy. Source histories match and normalized facts were verified.
WAL full-page images: 752 → 723. WAL is a server-wide snapshot sensitive to full-page images and background activity; it is not a stable amplification ratio. Before = PR base; after = tested commit. 🔺 marks increased cost. Concurrent report diagnostics use paced writers and roll back; that phase includes SQL and sleeps, not continuous SQL overlap or durable-commit throughput. Committed lifecycle samples include COMMIT; phase elapsed time and application thread CPU compare fixed logical work, not pipeline capacity. Concurrent diagnostic CPU is application process CPU. Statement WAL uses a server-wide LSN delta including AFTER triggers/background activity; unequal full-page images prevent a stable WAL-amplification estimate. Topic SQL replay totals come from separate EXPLAIN executions after each timed report. Backfill excludes migration DDL and lock waits. Committed topic measurements include commit latency; they do not measure full ingestion-pipeline capacity or establish an overall production benefit. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Feed-options scale comparisonCOMPLETE · measurement job: success. Baseline: 5a566acd4148 · Candidate: ae25dc11cc1b Full SQL plans, timing samples and job logs · attempt 1 Feed options benchmark: COMPLETE50/50 scenarios returned identical public JSON; 250 timed before/after pairs. Improvement = (before − after) / before × 100. + means less time/work; − means more. Default English: full options call
Source work at 1,000,000 articles, default English
Concurrent ingestion at 1,000,000 articles: 20,380 additional articles committed; 0 errors. All 50 route scenarios: before, after and signed improvement
Warmed query medians include SQL, ORM hydration, validation and JSON serialization. Each pair shares a read-only snapshot; order alternates. Source SQL metrics come from later EXPLAIN replay with supplied binds, separately from route timings and the exact prepared plans. This is synthetic evidence, not production HTTP p95. Unchanged content-type probes can still dominate absent-language requests; no speedup is claimed for every filter. Historical fixture loading bypasses USER triggers with FKs/indexes retained; measured ingestion runs migrated triggers. Shared ingestion measures contention, not before/after write amplification or capacity. |
Scheduler recheck scale resultsPASS · native ARM64 · PostgreSQL 17.11 · 5 × 50 rechecks + 50 concurrent ingests per dataset. Improvement: positive = less work or time; negative = regression. Before and after use identical historical populations. 100,000 articlesValid controlled comparison: native ARM64, PostgreSQL 17, matched dataset and workload.
Positive improvement means reduced cost; negative means increased cost. Delta = after - before. Base 150,000 articlesValid controlled comparison: native ARM64, PostgreSQL 17, matched dataset and workload.
Positive improvement means reduced cost; negative means increased cost. Delta = after - before. Base Measurements, query plans and job logs · attempt 1 Synthetic warmed scheduler/publisher workload; not production p95. Production reference: 92,581 articles on 2026-10-24. No deployment is performed. |
|
What changes and why?
Topic decision and inference reports expand and sort historical workflow JSON on each refresh. Project indexed, typed call facts transactionally from the existing run ledger so reporting reads matching calls without expanding historical evidence.
Each fact retains its original workflow position and lifetime stage occurrence. Completed-topic efficiency still includes the whole workflow, including calls outside the report window. Appends, settlements, corrections, removals, rollback and deletion follow the source transaction. Ongoing maintenance touches one workflow and has no global rollup or refresh job. Unchanged facts are filtered before the upsert, avoiding conflict locks on unaffected calls; a 20-call two-session regression fails with the previous trigger and passes with this fix.
The benchmark workflow updates one PR comment with separate read and write results, Before/After tables, signed cost deltas and improvement percentages. Positive improvement means reduced cost; negative means increased cost. Read gains do not imply an overall performance win. Separate committed-write comparisons cover append, settlement, correction and checkpoint transactions at both 5 and 20 lifetime calls. General profiling supports PostgreSQL 17+; the production comparison independently requires PostgreSQL 17. Disposable database credentials are generated at runtime, and profiling creates new JSON artifacts within
reports/without following symlinks or overwriting existing files.Related issues
Closes #299.
Validation
0036 → 0035 → headround trip verifies that topic source history and rebuilt facts survive while the scheduler triggers and clock-only projection updates remain active.bash scripts/ci/python-tests.sh integration 1 6).ae25dc11, including all six integration shards, every browser variant, CodeQL, SonarQube, all 12 ARM64 image build/scan/smoke jobs and CI required. SonarQube reports zero new issues and 95.8% new-code coverage. All native profile workflows pass their checks; the topic report independently discloses higher committed-write latency rather than claiming an overall performance win.Native ARM64 volume results
Mixed performance: topic reads improve; committed writes cost more. These are final-head
ae25dc11measurements from native ARM64 PostgreSQL 17.11 withwork_mem=4MB. The existing results comment updates in place and explicitly states read IMPROVED / write HIGHER LATENCY OBSERVED.Delta = after − before. Improvement = (before − after) / before; positive means reduced cost.
At 100k / 300k articles, fixtures contain 6k / 18k decision workflows, 18k / 54k calls and 500k / 1.5m inference rows. The read-only production reference had 92,581 articles, 5,000 workflows and 14,373 calls. Complete report JSON matched the previous implementation for 1/7/30/90-day windows at both sizes. Topic SQL replay was 12.0× / 23.6× faster and temporary writes fell from 55.69 MiB to zero.
Committed lifecycle costs at 300k articles and the 20-call lifetime limit:
Each 5- and 20-call cohort has 2,400 real commits per strategy across three alternating-order pairs. Latency includes COMMIT; final source states match and current facts are verified. The readable comment includes both dataset sizes and both lifetime limits, including negative improvements. The original rollback diagnostics remain separate.
WAL is server-wide and samples have unequal full-page images; no stable write-amplification percentage is established. Setup/cleanup is excluded and physical heap/index bloat is not restored between phases. These fixed lifecycle samples do not establish full ingestion capacity, sustained vacuum behavior or a net production performance win. No universal writer-latency budget has been defined.
Read diagnostics use four concurrent report workers, 12 report pairs and 200 paced rollback-only writer transactions. Writer-phase overlap includes pacing sleeps and does not prove continuous SQL overlap. Application process CPU for that phase changed 0.458 → 0.407 s / 0.505 → 0.466 s; this is not database CPU. Equivalent backfill took 0.412 / 1.295 seconds, excluding migration DDL and lock admission.
The separate public reader API comparison remains inconclusive; its passing CI threshold does not establish an improvement.
Rollout notes
Migration 0036 follows the scheduler migration 0035 from the current base. Backfill and trigger installation are atomic. Upgrade admission excludes existing run row lockers and releases partial table locks before retrying, failing cleanly after 30 seconds of sustained contention. Once admitted, proposal/run writes pause through the one-time backfill/index creation; ordinary reads continue.
Rollback application code and schema together. Downgrade uses bounded admission for all affected tables, draining writers and readers before its exclusive DDL locks. Downgrading to 0035 removes topic-derived facts/indexes/functions while preserving authoritative source JSON for rebuilding and retaining the scheduler clock capture.
Deployment is outside this task. After release, compare fresh query-statistics deltas under matched workloads and verify report freshness, worker latency, write amplification and correction/deletion recovery. See the benchmark and rollout guide.
Ready for review