Skip to content

Add frozen Slurm task plans and guarded packed execution - #213

Merged
jc-macdonald merged 1 commit into
mainfrom
feat/frozen-slurm-execution
Oct 20, 2026
Merged

jc-macdonald merged 1 commit into
mainfrom
feat/frozen-slurm-execution

Conversation

@jc-macdonald

@jc-macdonald jc-macdonald commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Queued jobs currently need project-specific scripts to keep their source commit, imports, packing and resume behavior fixed. Add an experimental explicit Python-task launch layer that freezes a detached checkout and records task argv, runtime/package versions, guarded import origins, data revision, resources and copied launcher identity.

Plans run locally or explicitly submit one CPU node allocation. Child tasks retain separate logs, locked atomic completion receipts and declared output hashes; missing outputs rerun the task, changed content/provenance is refused, and a failed child fails the batch. Active dependencies use afterok; completed dependencies require matching successful accounting records. Commands use argv, and the generated Bash bootstrap checks its copied launcher before execution.

This advances #174's minimal frozen execution contract. Existing grid ledgers and adaptive journals handle recovery inside explicit task commands. Direct executor= integration, automatic shard merging and a real target-cluster qualification remain open on #174; this PR does not claim a general distributed adaptive executor. No core dependency or existing runner behavior changes.

Validation: just ci passed (902 tests, 94.51% coverage), strict typing and strict MkDocs. Real Git/Python/grid-ledger tests cover changed submitting/frozen trees, stale imports, missing/corrupt artifacts, failed/resumed tasks and overlapping submissions. A scheduler fixture invokes the actual generated batch script through native Bash, checking dependency/accounting failures and CPU packing. A complete paired normal-mean example runs locally and prints an analytic comparison table; CI also executes it from the checkout and clean wheel.

The new optional public API warrants a minor release after merge; the pre-existing unreleased changes are documentation only. The package remains alpha, and cluster-specific accounting/locking/cancellation checks remain explicit deployment work.

Implemented with AI assistance.

Rockfish qualification used the wheel from final head 7ab4496, an isolated Python 3.11.9/NumPy 2.0 environment, and a shared-partition allocation of 2 CPUs, 1 GB, 2 minutes. Job 32273353 completed successfully (13 seconds); both grid tasks reproduced the documented analytic demonstration. Job 32273355 reused the same plan/receipts successfully (1 second), with a completed-job dependency. A dependency on failed job 32273322 was refused before submission. That initial job failed before task execution because its selected Python interpreter existed only on the login node; the shared Python module resolved the site configuration. This is a small single-site pilot, not general cluster or adaptive-executor qualification.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jc-macdonald
jc-macdonald merged commit 5a695c8 into main Oct 20, 2026
8 checks passed
@jc-macdonald
jc-macdonald deleted the feat/frozen-slurm-execution branch October 20, 2026 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant