Skip to content

feat: accept x- extension keys in native documents and run the OMG PSSM 1.0 test suite - #697

Merged
fgmacedo merged 6 commits into
mainfrom
fgmacedo/uml-w1_suite
Oct 14, 2026
Merged

fgmacedo merged 6 commits into
mainfrom
fgmacedo/uml-w1_suite

Conversation

@fgmacedo

@fgmacedo fgmacedo commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

Why

The UML work tracked in #692 needs a machine-checked reference: the OMG PSSM 1.0 test suite has 103 tests with an executable oracle in its XMI (ptc/18-11-06), and none of them run against the library today. This PR runs all 103 on both engines and records each divergence with a reason code. Refs #692.

What changes

  • tests/uml/pssm/convert.py turns the PSSM XMI into one native YAML case per test (cases/<Category>/<id>.yaml, 103 files, header "Do not edit"). It checks the sha256, is deterministic, and reads compiled activities only (Alf is read only for guard bodies). Each case is a native statechart that statemachine.io.load(path) loads as is; the test (stimulus, behaviors, expected traces, requires) lives under the x-pssm extension key. Constructs without a native spelling are listed in requires.
  • Native documents accept top-level x- extension keys: the parser and the JSON Schema ignore their content, any other unknown key is still rejected, and an x- key below the root too. Documented in docs/io/formats.md and in new 3.3.0 release notes.
  • Trace route: each trace() behavior becomes a bare callback reference. A harness listener appends the token, and event data reaches it by dependency injection. doActivities use the on_invoke_<state> convention; a doActivity that waits for an event parks until its state is left.
  • 10 hand-written overlays, keyed by XMI element id, cover behaviors with logic (counters, sends from inside the machine) and CallEvent tests that trace return or out values.
  • test_pssm_cases.py runs every case on the sync and async engines (206 runs). Any error fails the case loudly, including errors the async engine would otherwise drop after a doActivity completes.
  • conftest.py lists 44 expected failures with codes (requires:<construct>, conflict:K2/K10/K13/K14, bug:B1/B2, one unclassified). It also lists the 25 cases that pass only through the converter's lowering in LOWERED with lowered:<construct> reasons; those passes do not show native support. A test enforces that every case with a non-empty requires is classified.
  • test_pssm_drift.py regenerates the cases when PSSM_XMI is set and compares them byte for byte. The new pssm-drift CI job downloads the XMI with curl into a temp dir and runs that test with PSSM_REQUIRED=1; the XMI is never cached or committed.

Results, identical on both engines:

Category Cases Native pass Pass via lowering xfail
Behavior 5 2 1 2
Choice 5 0 4 1
Deferred 10 0 2 8
Entering 5 3 0 2
Entry 6 0 5 1
Event 16 14 0 2
Exit 3 0 0 3
Exiting 5 3 1 1
Final 1 1 0 0
Fork 2 0 1 1
History 8 3 0 5
Join 3 0 0 3
Junction 6 0 3 3
Other 1 0 0 1
Redefinition 6 0 6 0
Standalone 3 0 1 2
Terminate 3 0 1 2
Transition 15 8 0 7
Total 103 34 25 44

Evidence

$ timeout 180 uv run pytest -n 4
2919 passed, 26 skipped, 132 xfailed in 27.22s     # main without tests/uml: 24.9s (2 runs each, +2.6s)

$ timeout 120 uv run pytest tests/uml -n 4 -p no:randomly --no-cov
143 passed, 1 skipped, 88 xfailed in 1.41s          # stable over 10+ repeated runs

$ curl -fsSL -o /tmp/x/PSSM_TestSuite.xmi https://www.omg.org/spec/PSSM/20181101/PSSM_TestSuite.xmi
$ shasum -a 256 /tmp/x/PSSM_TestSuite.xmi
c355b249c356774377a46b60345019d827af1ce417bde88e533aa5f39206ae07
$ PSSM_XMI=/tmp/x/PSSM_TestSuite.xmi PSSM_REQUIRED=1 uv run pytest tests/uml/pssm/test_pssm_drift.py
1 passed

$ PYTHONHASHSEED=1 / 999 uv run python -m tests.uml.pssm.convert --out <dir>; diff -r <dir1> <dir2> tests/uml/pssm/cases
(no differences)

Under statemachine/, only the x- extension keys change (parser and schema). ruff, ruff format and mypy (pre-commit flags) are clean.

Adversarial review: four rounds, each by a fresh agent with no shared context.

  • Round 1 found 6 material issues, all fixed. Redefining transitions now take the union of their triggers. A doActivity cut at an accept no longer completes early (it parks). Async errors are caught. Reasons now list every blocker. A cleanup replaced document-order-changing YAML round-tripping with JSON, which turned two supposed bug:B5 cases into passes.
  • Round 2 found 1 material issue, fixed: an error while processing a doActivity's completion, in a step that leaves that state, was silently dropped on the async engine. The harness now records it, and a regression test covers it.
  • Round 3 found 1 material issue: strict=False on the lowered cases hid regressions. The coordinator decided to unmark them and list them in LOWERED.
  • Round 4 found nothing material. It cross-checked oracle and stimuli against an independent extraction for all 103 cases.

Not verified

  • The pssm-drift CI job has not run on GitHub Actions yet; it was emulated locally with the same curl command and env.
  • New finding, unclassified (History_002-B): completion transitions in two orthogonal regions, lowered to eventless, run in one microstep, while PSSM runs each completion in its own step. Not in the plan's K table.
  • New library finding, not given a code: on the async engine, InvokeManager._run_async_handler drops an exception raised while processing done.invoke when that same step exits the doActivity's state (ctx.cancelled is already set). The harness works around it; no PSSM case hits it today.
  • bug:B5 is not reproduced by any PSSM case after the key-order fix.
  • The OMG copyright sentence in tests/uml/pssm/README.md is generic (OMG plus the organizations named in the specification), written without quoting the PDF; the maintainer should check the wording.
  • The JSON Schema rejects a list target, while the parser accepts space-separated targets; the converter uses the latter for forks and entry points with several regions.
  • The harness reads the private sm._engine._invoke_manager state to know when doActivities have settled.
  • Overlay native actions run before the overlay's trace callback; current overlays do not depend on the order.

…t suite

Add a converter from the PSSM 1.0 test suite XMI (ptc/18-11-06) to native
YAML cases, the 103 generated cases, the overlays for behaviors with logic
and CallEvent tests, and a harness that runs every case on the sync and
async engines. Cases the library cannot pass yet are expected failures with
a reason code. A drift test regenerates the cases when PSSM_XMI points to the
XMI, and a separate CI job downloads it to run that test.

Refs #692

Signed-off-by: Fernando Macedo <fgmacedo@gmail.com>
@fgmacedo fgmacedo added this to the 3.3.0 milestone Oct 14, 2026
Comment thread .github/workflows/python-package.yml Fixed
Comment thread .github/workflows/python-package.yml Fixed
Comment thread .github/workflows/python-package.yml Fixed
Comment thread .github/workflows/python-package.yml Fixed
Comment thread .github/workflows/python-package.yml Outdated
- name: Compare the committed cases with the converter output
env:
PSSM_REQUIRED: "1"
run: uv run pytest tests/uml/pssm/test_pssm_drift.py --timeout 300

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not applied yet: PyYAML 6.0.3 publishes no cp315 wheels, so --no-build fails on the 3.15 jobs. Tracked in #684, which adds the flag once the wheels land.

Comment thread .github/workflows/python-package.yml Fixed
@codecov

codecov Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (d5282e0) to head (041fd96).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #697   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           53        53           
  Lines         5897      5901    +4     
  Branches       940       940           
=========================================
+ Hits          5897      5901    +4     
Flag Coverage Δ
unittests 100.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

… force HTTPS on the XMI download

Signed-off-by: Fernando Macedo <fgmacedo@gmail.com>
…ed accepts it

Signed-off-by: Fernando Macedo <fgmacedo@gmail.com>
cache-suffix: "python${{ matrix.python-version }}"
- name: Install the project
run: uv sync --all-extras --dev
run: uv sync --locked --all-extras --dev

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not applied yet: PyYAML 6.0.3 publishes no cp315 wheels, so --no-build fails on the 3.15 jobs. Tracked in #684.

enable-cache: true
cache-suffix: "pssm-drift"
- name: Install the project
run: uv sync --locked --all-extras --dev

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not applied yet: PyYAML 6.0.3 publishes no cp315 wheels, so --no-build fails on the 3.15 jobs. Tracked in #684.


- name: Install the project
run: uv sync --all-extras --dev
run: uv sync --locked --all-extras --dev

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not applied yet: PyYAML 6.0.3 publishes no cp315 wheels, so --no-build fails on the 3.15 jobs. Tracked in #684.

Signed-off-by: Fernando Macedo <fgmacedo@gmail.com>
Signed-off-by: Fernando Macedo <fgmacedo@gmail.com>
…-pssm

Signed-off-by: Fernando Macedo <fgmacedo@gmail.com>
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Security Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@fgmacedo fgmacedo changed the title test: run UML state machine cases generated from the OMG PSSM 1.0 test suite feat: accept x- extension keys in native documents and run the OMG PSSM 1.0 test suite Oct 14, 2026
@fgmacedo
fgmacedo merged commit 8be9e4b into main Oct 14, 2026
14 of 15 checks passed
@fgmacedo
fgmacedo deleted the fgmacedo/uml-w1_suite branch October 14, 2026 19:13
fgmacedo added a commit that referenced this pull request Oct 16, 2026
Lay out the 3.3.0 notes in the structure later PRs extend and document
the history default entry fix (#693), the done.invoke error reporting
fix (#698) and the PSSM-based UML test cases (#697).

Refs #692
Refs #713

Signed-off-by: Fernando Macedo <fgmacedo@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants