Repository navigation
Add frozen Slurm task plans and guarded packed execution - #213
Merged
Merged
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Queued jobs currently need project-specific scripts to keep their source commit, imports, packing and resume behavior fixed. Add an experimental explicit Python-task launch layer that freezes a detached checkout and records task argv, runtime/package versions, guarded import origins, data revision, resources and copied launcher identity.
Plans run locally or explicitly submit one CPU node allocation. Child tasks retain separate logs, locked atomic completion receipts and declared output hashes; missing outputs rerun the task, changed content/provenance is refused, and a failed child fails the batch. Active dependencies use afterok; completed dependencies require matching successful accounting records. Commands use argv, and the generated Bash bootstrap checks its copied launcher before execution.
This advances #174's minimal frozen execution contract. Existing grid ledgers and adaptive journals handle recovery inside explicit task commands. Direct
executor=integration, automatic shard merging and a real target-cluster qualification remain open on #174; this PR does not claim a general distributed adaptive executor. No core dependency or existing runner behavior changes.Validation:
just cipassed (902 tests, 94.51% coverage), strict typing and strict MkDocs. Real Git/Python/grid-ledger tests cover changed submitting/frozen trees, stale imports, missing/corrupt artifacts, failed/resumed tasks and overlapping submissions. A scheduler fixture invokes the actual generated batch script through native Bash, checking dependency/accounting failures and CPU packing. A complete paired normal-mean example runs locally and prints an analytic comparison table; CI also executes it from the checkout and clean wheel.The new optional public API warrants a minor release after merge; the pre-existing unreleased changes are documentation only. The package remains alpha, and cluster-specific accounting/locking/cancellation checks remain explicit deployment work.
Implemented with AI assistance.
Rockfish qualification used the wheel from final head
7ab4496, an isolated Python 3.11.9/NumPy 2.0 environment, and a shared-partition allocation of 2 CPUs, 1 GB, 2 minutes. Job 32273353 completed successfully (13 seconds); both grid tasks reproduced the documented analytic demonstration. Job 32273355 reused the same plan/receipts successfully (1 second), with a completed-job dependency. A dependency on failed job 32273322 was refused before submission. That initial job failed before task execution because its selected Python interpreter existed only on the login node; the shared Python module resolved the site configuration. This is a small single-site pilot, not general cluster or adaptive-executor qualification.