Repository navigation
Conversation
Backend unit coveragePASS · minimum combined coverage 98.00%. Tests: 10828 passed, 0 failed, 0 skipped. Source files: 367.
Unit coverage includes every backend workspace. PostgreSQL/Redis integration tests run separately. Commit: 9698777 · HTML, JSON, XML reports and job logs · attempt 1 |
Analysis retention benchmark · ✅ CompleteRun and raw reports · attempt 1 · commit 9698777d90c4 Benchmark job: success. Positive improvement = lower latency/cost; negative = regression. Native ARM64 / PostgreSQL 17. Decimal MB; benchmark observations are not production latency guarantees. 250,000 articles / 854,000 analysis jobs Idle and protected history
At the 80% latest-only cursor: at most 1,000 examined job rows/query, 5,667 shared buffers/query, 0 temporary blocks. Deployed cadence — five real 15-second scheduler ticks / 75 seconds
Ingestion transactions: 1,497 → 1,497. Payloads pruned: 1,000 → 600. Lower phase WAL includes the changed maintenance frequency; these are not equal-prune costs. Compressed stress — 50 consecutive pages
Payloads pruned: 10,000 → 10,000. This deliberately compresses scheduler spacing; use the deployed-cadence table for the sustained writer comparison. Fixture: 4.000 KiB full-history inputs; 64.000 KiB cadence input/catalog payloads on 1,000 subjects per history. The entire history does not use the larger payload. 100k results100,000 articles / 344,000 analysis jobs Idle and protected history
At the 80% latest-only cursor: at most 1,000 examined job rows/query, 5,107 shared buffers/query, 0 temporary blocks. Deployed cadence — five real 15-second scheduler ticks / 75 seconds
Ingestion transactions: 1,497 → 1,497. Payloads pruned: 1,000 → 600. Lower phase WAL includes the changed maintenance frequency; these are not equal-prune costs. Compressed stress — 50 consecutive pages
Payloads pruned: 10,000 → 10,000. This deliberately compresses scheduler spacing; use the deployed-cadence table for the sustained writer comparison. Fixture: 4.000 KiB full-history inputs; 64.000 KiB cadence input/catalog payloads on 1,000 subjects per history. The entire history does not use the larger payload. CPU is Python application process time including ingestion and the wait sampler, not database-server CPU. WAL includes ingestion. Each phase starts after an explicit checkpoint; automatic checkpoints remain enabled. |
PR performance resultsCurrent PR head: Token usage benchmarkPending: no results have been published. Reader API performanceSource head: PR reader API performanceResult: INCONCLUSIVE 5 inconclusive · 3 within threshold. Zero request failures across all six runs. Base Three independent GitHub-hosted ARM64 runners, each measuring base and PR sequentially with alternating order and isolated resources for every sample. 16 users, completed bootstrap and 15 seconds of warmup before each full 60-second measurement, fixed journey seed 294, 1,000 synthetic articles, cache off, five PgBouncer slots. Latency comparisonAll latency values are in milliseconds; lower is better. Values are medians of three runs, with one-millisecond histogram bins.
Additional latency and throughput metricsEach cell shows base → PR. Latencies are in milliseconds.
All paired p95 samples and changesEach pair shares a validated runner. Cells show base → PR p95 (absolute and percentage change), in milliseconds. Slower/faster counts use the same strict 20% and 20 ms thresholds against that pair's own baseline.
Individual runs
How to interpret this reportRegressed / Improved: a p95 regression needs >20% and >20 ms increase in the medians and in at least two same-runner base/PR pairs. Improvements use the inverse threshold. Within threshold means the change did not meet both thresholds. Inconclusive: a within-revision p95 range >35% of its median and >20 ms is too noisy for a confident latency verdict. A material median change without two confirming pairs is also inconclusive; this does not prove unchanged performance. Inconclusive latency comparisons are non-blocking: the check passes without claiming an improvement. Confirmed regressions and test/report errors fail the PR gate. Throughput is descriptive: paced users do not measure maximum capacity. This uncached public API test does not establish production health, browser performance or worker capacity. Base and PR share a runner within each pair; three pairs use independent runners. Alternating order reduces host-assignment bias but cannot eliminate changes in host load over time. Full HTML/CSV reports and logs · attempt 1 |
Lighthouse ·
|
| Page | FCP | LCP | CLS | TBT | JS | Total |
|---|---|---|---|---|---|---|
Latest (/latest) |
1.21 s | 3.25 s | 0.001 | 108 ms | 361 KiB | 663 KiB |
Article preview (/articles/…) |
1.37 s | 4.10 s | 0.005 | 273 ms | 364 KiB | 685 KiB |
Topics (/topics) |
1.21 s | 2.96 s | 0.000 | 102 ms | 319 KiB | 471 KiB |
Sources (/sources) |
1.22 s | 3.04 s | 0.000 | 282 ms | 332 KiB | 488 KiB |
Public profile (/users/…) |
1.22 s | 2.83 s | 0.009 | 89 ms | 280 KiB | 428 KiB |
Leaderboard (/leaderboard) |
1.07 s | 1.99 s | 0.000 | 74 ms | 276 KiB | 383 KiB |
Mobile reader · Three cold loads per page · Median timings · Largest transfer size.
Needs attention
| Page | Finding | Measured | Target / limit |
|---|---|---|---|
Latest (/latest) |
361 KiB | 330 KiB | |
Latest (/latest) |
3.25 s | 2.50 s | |
Article preview (/articles/…) |
364 KiB | 330 KiB | |
Article preview (/articles/…) |
4.10 s | 2.50 s | |
Article preview (/articles/…) |
273 ms | 200 ms | |
Topics (/topics) |
2.96 s | 2.50 s | |
Sources (/sources) |
332 KiB | 330 KiB | |
Sources (/sources) |
3.04 s | 2.50 s | |
Sources (/sources) |
282 ms | 200 ms | |
Public profile (/users/…) |
2.83 s | 2.50 s |
Full reports and logs · 9698777 · Attempt 1
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Confine generated reports to fixed paths held through validated file descriptors, generate disposable database credentials at runtime, and use reproducible SHAKE fixtures. Guard empty history pages explicitly and cover cursor, output isolation, and report freshness boundaries.
Set reader preferences through keyboard input and capture the emulated viewport. This avoids injecting mouse input or triggering Playwright full-page touch reapplication while retaining coarse-pointer, target-size, and native-tap assertions.
|
Feed-options scale comparisonCOMPLETE · measurement job: success. Baseline: 20b8fdf9f3b1 · Candidate: 9698777d90c4 Full SQL plans, timing samples and job logs · attempt 1 Feed options benchmark: COMPLETE50/50 scenarios returned identical public JSON; 250 timed before/after pairs. Improvement = (before − after) / before × 100. + means less time/work; − means more. Default English: full options call
Source work at 1,000,000 articles, default English
Concurrent ingestion at 1,000,000 articles: 20,460 additional articles committed; 0 errors. All 50 route scenarios: before, after and signed improvement
Warmed query medians include SQL, ORM hydration, validation and JSON serialization. Each pair shares a read-only snapshot; order alternates. Source SQL metrics come from later EXPLAIN replay with supplied binds, separately from route timings and the exact prepared plans. This is synthetic evidence, not production HTTP p95. Unchanged content-type probes can still dominate absent-language requests; no speedup is claimed for every filter. Historical fixture loading bypasses USER triggers with FKs/indexes retained; measured ingestion runs migrated triggers. Shared ingestion measures contention, not before/after write amplification or capacity. |
Scheduler recheck scale resultsPASS · native ARM64 · PostgreSQL 17.11 · 5 × 50 rechecks + 50 concurrent ingests per dataset. Improvement: positive = less work or time; negative = regression. Before and after use identical historical populations. 100,000 articlesValid controlled comparison: native ARM64, PostgreSQL 17, matched dataset and workload.
Positive improvement means reduced cost; negative means increased cost. Delta = after - before. Base 150,000 articlesValid controlled comparison: native ARM64, PostgreSQL 17, matched dataset and workload.
Positive improvement means reduced cost; negative means increased cost. Delta = after - before. Base Measurements, query plans and job logs · attempt 1 Synthetic warmed scheduler/publisher workload; not production p95. Production reference: 92,581 articles on 2026-10-24. No deployment is performed. |
What changes and why?
Retention searched old analysis history on every scheduler tick, spending about three seconds to select five payloads in issue #298.
Persist one sweep checkpoint per history. Each due page scans at most 1,000 identities and redacts at most 100 payloads, then waits 30 seconds; a completed sweep waits one hour. Ordinary ticks check only the two checkpoint rows. Redaction and progress commit together, so crashes and competing schedulers cannot lose the cursor.
Preserve the last retained successful result when newer attempts fail or remain active, alongside lease, age, provenance, review, verification and relationship protections. Lock jobs before rechecking eligibility in a fresh statement to close a verification-admission race. Partial successor indexes and explicit tuple bounds keep deep-page selection bounded. Parent locks permit ingestion foreign-key inserts.
Related issues
Closes #298.
Validation
9698777d: 465 Python regressions, 62 security/reporting tests and 23 reader-fixture/dispatch tests passed. This includes retention concurrency and page bounds, the real PostgreSQL 17.11 benchmark smoke, migration upgrade/downgrade and invalid-index retry, metadata comparison, and CI/Sonar contracts. Full Python checks, actionlint, formatting and commit hooks passed. Master20b8fdf9is included; retention is now migration 0037 after 0036, preserving master’s existing 0035/0036 migrations.reports/retention, with descriptor-held directories/files, link rejection and stale-report invalidation before database changes.aa0c1536: reader web, built Chrome and Edge extensions, plus their partner variants. Native-touch fixtures use keyboard setup and viewport capture to avoid injected mouse input and full-page touch reapplication. Coarse-pointer, 44 px target and native-tap assertions remain; no retry or assertion suppression was added.The full-cardinality fixtures use 4 KiB inputs; the separately paced backlog uses 64 KiB input/catalog payloads and 16 KiB article step outputs. WAL includes ingestion; application CPU includes writer/observer threads, not database-server CPU. These controlled pooled-connection measurements do not establish production latency. Benchmark controls and reproduction.
Rollout notes
Stop older schedulers, apply migration 0037, then start new runtimes. The migration seeds two checkpoint rows and creates three concurrent partial indexes; it does not redact payloads. Retention caps statements at two seconds, lock waits at 100 milliseconds and PostgreSQL 17+ transactions at five seconds.
Migration retries repair invalid new indexes while preserving committed checkpoints. Rollback requires stopping new runtimes and downgrading to 0036 before starting older binaries. Pruned input/catalog/step-output bodies cannot be restored; identities, results, hashes and accounting remain.
Defaults allow at most 288,000 prunes/history/day before protected scans, cooldown and scheduler delay. Fresh production successes were 5,499 article/4,277 topic jobs over the trailing 24 hours. Monitor actual progress rather than treating the ceiling as achieved throughput.
Production has only been read for bounded sizing evidence. After deployment, compare matched workload windows for latency, ingestion freshness, WAL and interrupted-sweep recovery. Operator and rollback details.
Ready for review