Repository navigation
Conversation
A process is identified by the SHA-256 of a canonical form defined over the parsed JSON document, not over any CWL library's output, so that it is deterministic, stable across library upgrades and reproducible by any tool. Ids, doc and label where they are metadata, $namespaces and dirac: hints are dropped; prefixes are expanded to full URIs; lists become maps; fully scoped identifiers are shortened against their scope; type shorthands are expanded and named types written by short name. Each child run body is replaced by its own identity, so a workflow's identity covers its children's (a Merkle tree). pack and unpack convert to and from a self-contained $graph whose components are named by their identity. Documents must be resolved before canonicalisation: $import, $include and any other schema-salad directive is refused, since the identity would not cover what it names and a runner would fetch it. Standard library only. Closes DIRACGrid#1106 Part of DIRACGrid#1080
Unit tests ran on SQLite only, but production runs on MySQL and MariaDB, which differ from SQLite where the transformation tables rely on them. The diracx-db transformation tests now share one fixture that runs each test on SQLite and on every server given by DIRACX_TEST_MYSQL_URL or DIRACX_TEST_MARIADB_URL, and a CI job runs them on MySQL 8.4 and MariaDB 11.4 service containers, pinned by digest. It found one difference already: a failed CHECK constraint is an IntegrityError on SQLite but an OperationalError on MySQL and MariaDB through PyMySQL, so the deltas test now accepts either, matched on the constraint's name. Closes DIRACGrid#1108
…onDB insert_processes stores documents under their content hash with INSERT IGNORE, so concurrent submissions of a shared process are safe, and get_processes fetches them, on the DX-ADR-004 Processes table unchanged. MySQL's JSON type re-sorts keys and reformats numbers, so a stored row is not the bytes that were hashed: what must hold is that the value read back re-hashes to its identity, which the round-trip test checks on SQLite, MySQL and MariaDB. Closes DIRACGrid#1109 Part of DIRACGrid#1080
check_policy checks every process of a canonical tree against rules beyond valid CWL, and lists every violation in one error: - requirements outside an allow-list every backend can honour are refused, on a process and on a step, dirac: requirements included; hints are not restricted; - DockerRequirement is accepted, to run with Apptainer, for an image to pull (dockerPull, dockerImageId); dockerFile, dockerLoad and dockerImport are refused, in hints too, since they build or fetch an image on the worker node; - abstract Operations are refused, since no runner can execute them; - processes larger than 256 kB are refused. Closes DIRACGrid#1107
store_process canonicalises a submitted process, checks the policy, and returns at once if it is already stored. Otherwise it inlines any stored process it references, packs the tree into one self-contained $graph, validates every component of it against the CWL v1.2 schema with cwl-utils and stores it: one Processes row per transformation, keyed by the Merkle root, so a parcel's payload is a single read (load_process) and exactly what a stock runner executes. Validation reads nothing but the packed document: any other file or URL is refused rather than fetched on the server. Step input wiring is not checked yet, until a cwl-utils release fixes its static checker (common-workflow-language/cwl-utils#480). The MCSimulation fixture is the JSON the DIRACGrid#1080 workgraph resolves to, as the submission endpoint will receive it. Closes DIRACGrid#1110 Part of DIRACGrid#1080
For a hotfix, such as a new software version, an operator can make a paused transformation run another process. End users cannot: it needs the ProductionManagement or Operator property, and no submission API offers it. A stored process never changes, since its key is its hash; what changes is which process the transformation points at. It refuses unless the transformation is Paused (its row is locked for the change), a reason is given, the replacement passes the same policy and validation as any submission, and it keeps the interface of the process it replaces (input and output names and types, named types compared by definition), since the feeder, the packer, the declared outputs and downstream edges depend on it. In the caller's transaction it then repoints the transformation and its Unassigned parcels, and records ProcessReplaced in TransformationLog. From Reserved on, a parcel keeps the process it was created with, which is why ComputeParcels.ProcessID must stay a column of every compute parcel: once a transformation's process can change, a parcel's own copy is the only record of what it ran. DX-ADR-004 freezes a transformation's ProcessID once it has parcels, and DX-ADR-005 cancels and resubmits a wrong configuration. Both still hold for owners; this is the operator's exception. Closes DIRACGrid#1111
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1111.
Blocked by #1118, whose commits this branch includes until they merge.