Skip to content

fix(package): write packages.json pages in a deterministic order - #188

Open
LukasGold wants to merge 1 commit into
mainfrom
fix/172-package-page-order
Open

LukasGold wants to merge 1 commit into
mainfrom
fix/172-package-page-order

Conversation

@LukasGold

@LukasGold LukasGold commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

⚠️ Read before merging: downstream repositories need a one-time review

This change alters the order of the pages array in every generated
packages.json. The first rebuild of a downstream package repository after
upgrading produces one large diff, potentially several hundred lines, even
when no wiki content changed at all. Every rebuild after that produces a small
diff, which is the point of the fix.

That one-time diff has to be reviewed, because a real content change could hide
inside it. Do that review AI-assisted, not by reading the lines. A human
reading several hundred reordered lines will not reliably spot a single changed
slot.

Ask an agent to compare the two packages.json revisions semantically rather
than textually: parse both files, then report the set of page names added and
removed, the pages whose slot content differs, and any change to the package
metadata. The issue this PR closes already shows what that output looks like:
"1 page added, 0 removed, 0 content-changed, no metadata change, but 743
changed lines."
A clean result is every category empty except the intended
change.

No packages.json is committed in this repository, so nothing here changes.

Closes #172

Cause

The issue attributes the churn to the parallel page fetch appending attachments in completion order. That is not the mechanism: file_dumps is only read by key at src/osw/wtsite.py:1005 and :1007, never iterated, so its insertion order never reaches the output. There are three real sources:

  1. src/osw/wtsite.py:2228 - find_file_page_refs_in_slots returns list(set(file_page_refs)). CPython randomizes string hashes per process, so this order varies between runs.
  2. src/osw/wtsite.py:967-972 - page_files[page.title] = list(set(referenced_file_pages) - set(added_file_titles)). A second set round-trip. This is the list the final assembly loop walks, so this alone explains the measured churn.
  3. src/osw/wtsite.py:919-942 - pages is filled by pages.append inside a thread pool (:417, via parallelize at :446). The loop at :947 accumulates added_file_titles while walking pages, so when two configured pages reference the same file, which page claims it varies per run and the file moves in the output.

Source 3 also appears without parallel fetching: src/osw/wtsite.py:926-928 does list(set(titles_to_fetch) - set(param.offline_pages.keys())), so the fetched pages arrive in set order whenever offline_pages is supplied.

Changes

  • page_files[page.title] now uses sorted() instead of list(). Makes each page's attachments alphabetical by File: title and neutralizes sources 1 and 2. This is the fix proposed in the issue.
  • pages is sorted by rank in added_titles before the loop that claims files. Fixes source 3. added_titles comes from list(dict.fromkeys(config.titles)) and is already deterministic.
  • Two offline regression tests in tests/test_wtsite_create_page_package.py.

The commented-out bundle.packages[config.name].pages.sort(key=lambda x: x.urlPath) at src/osw/wtsite.py:1010 is left as is. Enabling it would reorder the Item: entries by urlPath and lose their correspondence with config.titles, as the issue notes.

find_file_page_refs_in_slots is unchanged. It is public API, and the sorted() change makes its order irrelevant to this output.

Rationale for the shared-file rule

When several pages reference the same file, it is now always attached to the page that comes first in config.titles. This is not a new rule: the existing - set(added_file_titles) filter at src/osw/wtsite.py:971 already means the first page to reference a file wins. The change makes that rule deterministic instead of dependent on thread completion order.

Verification

  • Both new tests fail against the unfixed source and pass with it, under PYTHONHASHSEED 0, 1, 42, 12345 and 99999.
  • Full non-integration suite: 301 passed, 30 skipped, 0 failures.
  • ruff check and ruff format --check clean on both changed files.

Not verified against a live wiki; I had no credentials. Two consecutive builds of world.opensemantic.meta.docs would confirm the checksum is now stable.

Known limit of the tests

The tests substitute WtPage.dump with a stand-in returning only name, namespace and slots, and compare only the name sequence. That is sufficient for an ordering test, because the change never alters a dump's contents, only which dump object goes where. The tests would be blind only if two entries shared a name.

- sort each page's file attachments instead of leaking set iteration order
- restore the configured page order before the loop that claims files
- keep Item: entries aligned with config.titles, no full-array sort
- add two offline regression tests covering both ordering sources

Closes #172
@github-actions

Copy link
Copy Markdown
Contributor

Release preview

Merging this PR would release v2.3.6 (current: v2.3.5).

Changelog preview (truncated)
## v2.3.6 (2026-09-21)

### Bug Fixes

- **package**: Write packages.json pages in a deterministic order
  ([`467d11c`](https://github.com/OpenSemanticLab/osw-python/commit/467d11cc26a0a94f3a31936124740f33c31692c0))

Preview via python-semantic-release and conventional commits.

@LukasGold LukasGold self-assigned this Sep 21, 2026
@LukasGold LukasGold added the enhancement New feature or request label Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

packages.json page order is not deterministic for auto-resolved file attachments

1 participant