Repository navigation
Feat/factors - #20
Merged
Merged
Feat/factors#20
Conversation
added 30 commits
October 6, 2026 15:35
added 27 commits
October 13, 2026 08:49
Row-level result triggers re-updated the same result_refreshes row for every changed row, so bulk writes slowed quadratically (100k rows: 8+ min -> 28s).
The Cells tab loaded every replicate (with artifacts) to average them in the
browser. GET /experiments/{id}/cell-summaries returns one row per cell with
metric sum/count, so the payload scales with factor combinations.
…rowser DatasetRowResults/Runs/Cells and StopRowRuns had no importers left and read the whole-experiment run-results payload; their tests only exercised that path.
ASAREE queried core's tables directly and created its own indexes on them via a one-off compose job. Motoro now owns those indexes and exposes list_run_summaries/list_run_step_summaries; use them and drop asaree-results-index. Requires the Motoro release containing them.
…ro's runner read_prompt, read_step and the compatibility usage lookup queried core's tables through system_session. Use read_run_input_chunk, read_run_step_chunk and list_run_summaries instead; ASAREE no longer queries Motoro tables directly.
Follow migrations/guards.py's convention: guard every create/drop with IF [NOT] EXISTS, CREATE OR REPLACE the functions, and drop each trigger before recreating it. Verified: main -> head on fresh and real data, a full re-run leaves the schema identical, and each downgrade survives a lost stamp.
Provides list_run_summaries, list_run_step_summaries, read_run_input_chunk and read_run_step_chunk, which ASAREE's results reads now depend on.
…ping tests
Shared-resource groups were labelled in connection order ("Zebra, Alpha") while
every other picker level sorts by name; the same label feeds each factor
dialog's "Applies to" line. The tests still expected the grouping from before
shared resources got their own group and the agent connector was renamed.
Opt-in (ASAREE_MIGRATIONS_TEST_DATABASE_URL) check, modelled on Motoro's, that each revision re-runs cleanly over its own result, in both directions, with an identical schema. Eight upgrade and nine downgrade revisions that predate it are listed as known exceptions and must keep failing until they're guarded.
Nine revisions failed when re-run over their own result (upgrade or downgrade): unguarded DDL, and data steps reading what the first run had already dropped. Add guards.create_unique_constraint, has_table and has_column, guard the DDL, condition each backfill on the shape it reads, and let the a7b8 rename-and-normalize detect that it already ran. A fresh upgrade to main's head produces a byte-identical schema; the re-run test now enforces every revision with no exceptions.
ASAREE_RESULTS_TEST_DATABASE_URL and ASAREE_MIGRATIONS_TEST_DATABASE_URL were never set, so the result-read and migration re-run checks always skipped, and the frontend job never ran vitest.
A one-off db-backup compose service runs before motoro-migrate. When a
database has a pending migration it writes
backups/{db}-pre-v{release}-{revision}-{UTC}.dump (pg_dump -Fc, owner-only,
written atomically), keeps the newest five per database, never touches manual
dumps, and fails the deploy if the dump fails. Also keep backups/ out of the
Docker build context.
An attempt that hit its iteration limit finished with a (truncated) result, so "Run all" now skips it like a completed one instead of re-billing it. Each one gets a "run again" checkbox in the batch dialog and is covered by "run every completed execution again". One rule (RERUN_ON_REQUEST_STATUSES) now drives the batch planner, replicate planning, row claims and the retry endpoint, which previously disagreed between per-row and per-replicate runs.
Code passed as a one-off with no dataset workspace was written to inline-<hash>.py in the server's cwd and never removed, so it piled up in the deployed container and the test suite left one in the repo root on every run. Write it to a temporary directory removed after the run; the script still runs in the same working directory, so files it leaves for its next call are unaffected.
randomUUID only exists in secure contexts, so the Runs tab crashed when the app was opened over plain HTTP via a LAN IP or hostname. Build the key from crypto.getRandomValues, which has no such restriction, and keep it fully random since a repeated key returns an earlier job.
They rendered on every experiment, so an idle one offered to stop runs that didn't exist. Follow the summary's has_active and each cell's pending/running/finalizing counts, like the per-replicate stop button already does, and keep a clicked stop visible while its cancellation runs.
{db}-{revision}-before-v{release}-{UTC}.dump reads as what the file holds
(that schema revision) and when it was taken (before that release migrated
it), instead of pre-v{release}, which looked wrong when the release was
already installed.
…plicate generate_design_cells called upsert_replicate per planned replicate, and each call reloaded every replicate already in its cell, so a 6 x 2,631 design took about 4.5 minutes and timed out through the SDK and UI. materialize_replicates applies the same rules (cell by factors, then label; merges; published revisions frozen) after reading the revision once, and writes every new row in one flush: 15,786 replicates generate and regenerate in about 1.4s. The endpoint also takes include_replicates=false, which the canvas uses, to skip listing every replicate back; the SDK default still returns the list. Also import protocol_run in test_design_revisions so the file runs alone.
db-backup ran as a fixed UID 1000, so on a Linux host whose user has any other UID it couldn't write ./backups and, by design, stopped the deploy. Run it as root and chown each dump to the owner of the mounted directory; Docker Desktop on macOS maps ownership itself, so a refused chown is ignored. ASAREE_BACKUP_UID/GID are gone.
arq 0.28 has no _keep_result enqueue option, so it passed it to execute_result_export as a keyword argument and every export crashed on start; the UI sat on "preparing..." forever. The worker already sets keep_result=0. The test checks every enqueue option against arq's own signature.
The background export (added on this branch) wrote a raw dump: one column per stored field, with tokens, cost, duration and metrics inside JSON. Researchers lost the analysis CSV main produced. The job now writes that CSV: the Results projection runs a chunk of replicates at a time (new replicate_ids filter), pass one lays out columns from compact stubs, pass two writes rows in factor-level order, all in one REPEATABLE READ snapshot. The CSV writer is split into a shared layout so both downloads are produced by the same code; byte-identical to the synchronous CSV on all 102 published dev experiments. 15,786 runs: 22s, ~110 MB peak instead of 740 MB for the whole projection.
get_db commits when FastAPI tears the dependency down, which happens after the response is sent (FastAPI 0.141), so creating a batch preview and polling it straight away could return job_not_found -- the SDK hit it on Run cell. Create, confirm and cancel now commit before answering.
…g batches FOR UPDATE also blocks foreign-key checks on the row. Runs finishing during a large Run all write the results trigger's refresh row (which references the protocol) while the planner held the protocol row and waited on that refresh row: 2 of 485 runs died with a deadlock. FOR NO KEY UPDATE still serializes planners, Stop and publishing, but lets those checks through.
A preview step checked 100 replicates (per row: one replicate) and the worker ran one step per one-second tick, so previewing a 15,786-replicate experiment took 2.7 minutes and a per-row design with thousands of replicates would take most of an hour; queueing confirmed runs crawled the same way. Steps now check up to 1,000 replicates (per row: as many whole replicates as fit in 1,000 rows), and each tick keeps advancing a job for up to 0.9s. Every step is still its own short transaction, so progress, cancellation and locking are unchanged.
slot_query and the rollup joined each slot's latest run on replicate_result_id = x OR (<legacy> AND ...), with <legacy> a bound parameter. Once a reused connection switched to a generic plan, Postgres couldn't fold the OR and scanned every run of the design for each slot: rebuilding a 15,786-run experiment's summary took 64 s instead of 0.37 s, mid-batch and repeatedly. legacy_flag() and true()/false() render constants, which generic plans fold. A test rejects bound booleans in these statements.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Factor controls for every canvas connector, plus a rework of how results scale. A user with a 15,786-run experiment couldn't open the Runs or Results tabs: the request ran for 5+ minutes and ended in a Bad Gateway. Both tabs now open in under two seconds at that size, and every bulk path (Run all, Stop, export, design generation, summaries) was fixed for the same scale.
Pairs with Motoro v0.12.0 (EpistasisLab/motoro#12), which is required.
Why the tabs timed out
Runs and Results loaded the whole experiment in one request, every run with its full outputs and traces, and re-polled every 3–5 seconds. Fixing that exposed the same "load everything" pattern in design generation, the Cells tab, CSV export and batch planning, so each of those changed too.
What changed
Factorial design
Results reads
/execution-results/...) plus a precomputed summary, maintained by a new results worker. Database triggers mark which summaries are stale.GET /experiments/{id}/cell-summaries).Bulk operations as durable background jobs
Performance fixes found at scale
FOR NO KEY UPDATE, which removes a deadlock between the batch planner and runs finishing mid-batch.Motoro boundary
list_run_summaries,list_run_step_summaries,read_run_input_chunk,read_run_step_chunk).Operations
db-backupcompose service: before migrations, dumps each database with a pending migration tobackups/{db}-{revision}-before-v{release}-{UTC}.dump(newest 5 kept; owned by the directory's owner).tests/test_migration_rerun.pyenforces it.ASAREE_RESULTS_TEST_DATABASE_URL,ASAREE_MIGRATIONS_TEST_DATABASE_URL) and vitest.Smaller fixes
crypto.randomUUID); Stop buttons shown only while something is active; export jobs never starting (_keep_result); job endpoints commit before responding (read-after-create 404); one-off scripts no longer leaveinline-*.pyin the working directory; factor picker tests updated.Behaviour changes
limit_reached) executions unless they're ticked to run again, the same as completed ones.Testing
tsc -b, ruff and oxlint clean.main→ head verified on fresh databases and on copies of real dev data; first-run schema identical tomain's.Deploy notes
docker compose up -d --build --remove-orphans. Order:db-backup→ Motoro migrations → ASAREE migrations (5 new) → app and workers.asaree-results-worker. Without it, summaries stay "preparing", and Run all / Download CSV never finish.Review guide
services/result_jobs.py(what gets planned and billed),services/result_reads.py(slot_query), andservices/result_exports.py.RESULTS_OPERATIONS.md.