Skip to content

Render map wikitext and read GeoJSON pages with the page's parser - #951

Draft
JeroenDeDauw wants to merge 17 commits into
masterfrom
fix/render-on-page-parser
Draft

JeroenDeDauw wants to merge 17 commits into
masterfrom
fix/render-on-page-parser

Conversation

@JeroenDeDauw

@JeroenDeDauw JeroenDeDauw commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Maps rendered map wikitext, such as marker text and popup templates, and read GeoJSON pages outside the parse of the
page the map is on:

  • Marker text, inline labels and shape text went through Parser::parse() on a copy of the page's parser, or of
    MediaWiki's main parser for the semantic map formats. What that parse recorded, such as templates, links, categories
    and used parser options, went to an output that Maps discarded. parse() also moved the usage tracking of the shared
    parser options onto that output, so options used after a map no longer varied the page's parser cache.
  • The semantic map formats expanded template= popups on a parser of their own, with the requesting user's parser
    options, whose use the page's output did not record. So the popups in a cached render depended on whoever caused
    that render.
  • GeoJSON pages were read from the revision store, not through the parser.

So editing a template or GeoJSON page that a map uses did not refresh the page, links in map wikitext were missing from
the link tables, and readers could get a page with a map rendered for other settings, such as another interface
language. The template and current-revision callbacks of the parser options, which extensions use to decide which pages
a render may include, could not vary the page's cache by what they decided for marker text, applied to template=
popups for the requesting user, and did not apply to GeoJSON pages at all.

Maps now parses each piece of map wikitext with a copy of the page parser's options, on a parser that each map gets from
the parser factory and that starts a fresh parse for each text. The semantic map formats get the page's parser from
Semantic MediaWiki's copyParser(): for #ask and #show, the parser running the query. A parse per text keeps
rendering time linear in the number of markers: parsing each text on the page's own parser makes core render every link
collected so far again, taking 5.8 s for 800 linked markers. A copy of the page's parser would share the state that
extensions keep on it, which starting a parse resets: on MediaWiki 1.43 and 1.44, the page's own TemplateStyles styles
then went missing after a map that used the same styles.

Core's mergeInternalMetaDataFrom() and mergeTrackingMetaDataFrom() then add what the parse recorded to the page's
output: used options, flags, templates, links, categories and extension data sets, such as the pages a callback left
out. The parse's head items, config vars, modules and styles go along through mergeHtmlMetaDataFrom() of an output
that holds only these, so the title, indicators, table of contents and index policy stay the page's. Left out is what
the page decides: page properties and the flags of behavior switches, such as the default sort key and the index policy,
and extension data without a merge strategy, which the merge would replace. That is where extensions such as Semantic
MediaWiki, PageImages, Widgets and Kartographer collect data about the page. A ParserOutput hands over all of its
extension data only in its serialization, so Maps reads the keys from there. MediaWiki 1.46 marks the three merge
methods @internal, as used only by RevisionRenderer, but no public method adds one output to another.

Strip markers in map wikitext belong to the page's parser, so their content goes along to the parser of the text, each
as its own kind, also when nested in another. Tags that parse wikitext, such as galleries, leave link placeholders in
their HTML that only the page's parser can fill in, so the page's parser renders such strip markers with
recursiveTagParseFully() first. When the page's parser only preprocesses, as it does for parser functions under
Parsoid, extension tags in map wikitext are left unprocessed, so Maps puts their wikitext back and parses the map
wikitext in HTML mode; for MediaWiki 1.44 and later, see the last item below. The Widgets extension fills in its widgets
when the parse of the page ends, by which time the markers of a text sit in the map's data, so Maps fills in the widgets
of each text, and those the page expanded in the arguments of the map, before the text goes into the map's data.

GeoJSON pages are read as Scribunto's mw.title:getContent() reads pages, with the page parser's
fetchCurrentRevisionRecordOfTitle(), so the current-revision callback applies to them. Every existing GeoJSON page
Maps reads is recorded as a template of the page, even when a callback refuses it: the link tables come from one render,
but an edit of the GeoJSON page must also reach the renders that showed it.

On Special:Ask and Special:CargoQuery, which nothing caches, Maps uses a parser of its own for the requesting user, as
the semantic map formats did for popups before. Where it gets no parser that is parsing, as when Semantic MediaWiki
hands over one that never parsed or whose parse is over, when an older Cargo hands over a ParserOutput, or when a
caller of MapsFactory::newGeoJsonFetcher() passes none, it parses for an anonymous reader, because it cannot tell
whether the output ends up in a render cached for everyone.

Visible changes:

  • Categories and language links in map wikitext and popup templates now apply to the page, so category queries, such
    as those of Semantic MediaWiki, find it. Maps applied the categories of popup templates before 12.1.0, when it
    expanded them on MediaWiki's main parser.
  • {{PAGENAME}} and relative links in popup templates refer to the page instead of Badtitle/Parser
  • <nowiki>, <pre> and other tags in {{#display_map:}} map wikitext render instead of showing as strip markers,
    except under Parsoid on MediaWiki 1.44 and later, as described below
  • On MediaWiki 1.43 and 1.44, the page's own TemplateStyles styles no longer go missing after a <display_map> whose
    marker text uses the same styles
  • The content of references in <display_map> marker text shows nowhere, as hooks for the end of a page's parse no
    longer run for each text and so the popup gets no reference list. References in popup templates are numbered per
    popup rather than across the map. The page itself numbers and lists those in {{#display_map:}}
  • Widgets render in {{#display_map:}} marker text and in the popups of semantic maps, where they showed as raw
    markers
  • With Semantic MediaWiki 7.0 and older, charts and other query results above a map whose markers have a title or text
    keep their data, scripts and styles, which the parse of map wikitext used to take

Rendering {{#display_map:}} takes as long as on master. Seconds on MediaWiki 1.43:

Markers Each links a page On master No links On master
100 0.05 0.05 0.04 0.04
400 0.18 0.21 0.15 0.17
800 0.38 0.38 0.31 0.31
6400 3.88 3.70 2.66 2.80

1600 markers after 300 Semantic MediaWiki subobjects took 1.11 s, against 1.19 s on master.

Popups of the semantic map formats, which master expanded on one parse state, now get a parse each. Seconds for 500
popups, with LuaSandbox:

Popup text Branch Master
Links a page 0.12 1.11
No links 0.07 0.04
Calls a Lua module 1.41 0.12

On master, the time for linking popups grows with the square of their number: 2000 took 17.7 s. Lua starts anew for
each popup, as it already did for marker text on master, and each popup has the limits of a parse of its own, including
Lua's CPU time limit, so a large map whose popups run heavy Lua can reach the request timeout.

New tests cover each problem above. Compared before and after with a script, the output of 15 representative maps is
otherwise the same: 14 on a page, rendered by the legacy parser and by Parsoid, and one on Special:Ask. The Cargo
format, which uses the parser only to read GeoJSON pages, is unchecked: Cargo was not installed.

CI now also runs on MediaWiki 1.46 and, allowed to fail, on MediaWiki master, which renders with Parsoid by default.
Their composer-generated PHPUnit configuration fails tests without a @covers annotation, which many Maps tests lack,
so these jobs relax that, as core's former tests/phpunit/phpunit.php entry point did.

Public method signatures change: MappingService::newMapDataFromProcessingResult() and newMapDataFromParameters(),
CargoOutputBuilder::buildOutputFromCargoData(), QueryHandler::setTemplate() and the GeoJsonFetcher constructor now
take the parser rendering the page, the WikitextParser constructor also takes a ParserFactory, and MapsFactory
gains getRenderingParser(), newRequestParser() and newWikitextParser(). None of these has a known caller outside
Maps. The one Maps method that Semantic Result Formats calls, MapsFactory::newGeoJsonFetcher(), takes the parser
optionally. Hence the release notes list this under 15.1.0 rather than a new major version.

Considered, omitted:

  • The MediaWiki:Maps page: site configuration, read once per request and documented to take effect at the next parse
  • Icons, image layers and overlays, and KML files: Maps looks up their file URLs, not page content
  • A dependency on GeoJSON pages that do not exist: URL sources would then show up as wanted pages. Creating such a page
    still does not refresh the maps that name it
  • Reading GeoJSON pages through the template callback, as transclusion does: it follows redirects and hands over text
    rather than the revision, whose content model and ID the GeoJSON editor needs
  • Page properties, the display title, indicators and behavior switches in map wikitext: as before, they do not apply to
    the page, apart from the tracking categories some switches add, such as for __NOINDEX__. The page expands the
    arguments of {{#display_map:}} itself, so a {{DEFAULTSORT}} there still applies
  • Semantic MediaWiki annotations in map wikitext: as before, they do not annotate the page
  • The <!--MW-PAGEIMAGES-CANDIDATE-N--> comments that PageImages adds to images stay in popup HTML, where nothing shows
    them
  • Extra parser cache keys that extensions add with ParserOptions::addExtraKey() while parsing a text: the parse uses a
    copy of the page's options, so the keys do not vary the page's cache
  • Moving the hand-over of strip markers to a pull request of its own: parsing a text with recursiveTagParseFully()
    rather than parse() stops defusing the page's strip markers, so this pull request has to hand them over, and the
    Parsoid sanitization lives in the same step
  • Semantic Compound Queries and other callers that get whichever parser Semantic MediaWiki last held: while that parser
    is parsing another page, what map wikitext records lands in that page's output, where before it went to a discarded
    copy
  • Semantic Result Formats' filtered format, which calls newGeoJsonFetcher() without a parser and so reads GeoJSON
    pages as an anonymous reader: passing its parser is a change in that extension
  • Extension tags in {{#display_map:}} map wikitext under Parsoid on MediaWiki 1.44 and later: core leaves them for
    Parsoid to run, so <nowiki> content is parsed as wikitext, where before it showed as a strip marker, and other tags
    still show as strip markers, as before. Telling these strip markers apart takes StripState methods that MediaWiki
    1.43 lacks

AI-authored — Claude Code, Opus 5.5 (max); detailed spec from @JeroenDeDauw, rendering reworked once on an orchestrator decision and revised after AI code, test and prose reviews; diff not yet human-reviewed; regression tests failed before each fix, PHPUnit with SMW 7, phpcs and PHPStan pass locally on MediaWiki 1.43, map output and timings compared with master on the legacy parser and Parsoid, CI green with SMW 6 and 7.

JeroenDeDauw and others added 17 commits October 13, 2026 22:41
Maps rendered map wikitext and read GeoJSON pages outside the parse of the page the map is on:

* Marker text, inline labels and shape text went through `Parser::parse()` on a copy of the page's parser, or of
  MediaWiki's main parser for the semantic map formats. What that parse recorded, such as templates, links, categories
  and used parser options, went to an output that Maps discarded. `parse()` also moved the usage tracking of the shared
  parser options onto that output, so options used after a map no longer varied the page's parser cache.
* The semantic map formats expanded `template=` popups on a parser of their own, with the requesting user's parser
  options, whose use the page's output did not record. So the popups in a cached render depended on whoever caused
  that render.
* GeoJSON pages were read from the revision store, not through the parser.

So editing a template or GeoJSON page that a map uses did not refresh the page, and links in map wikitext were missing
from the link tables. And the template and current-revision callbacks of the parser options, which extensions use to
decide which pages a render may include, did not reliably apply to map wikitext and GeoJSON pages.

Maps now renders map wikitext with `recursiveTagParseFully()` on the parser of the page, which the semantic map formats
get from Semantic MediaWiki's `copyParser()`: for `#ask` and `#show`, the parser running the query. It reads GeoJSON
pages with that parser's `fetchCurrentRevisionRecordOfTitle()`, and records every existing GeoJSON page it reads as a
template of the page, also when a callback refuses it: the link tables come from one render, and an edit of the GeoJSON
page must also reach the renders that showed it. Where Maps has no parser that is rendering the page (on Special:Ask and
Special:CargoQuery, when Semantic MediaWiki hands over a parser that has not parsed yet, and for
`MapsFactory::newGeoJsonFetcher()` callers that pass none), it uses a parser of its own for an anonymous reader, because
it cannot tell whether the output ends up in a render cached for everyone.

Visible changes:
* Categories in marker text and popup templates now apply to the page, as they did for popup templates before 12.1.0,
  when Maps expanded them on MediaWiki's main parser. So do Semantic MediaWiki annotations in marker text; those in
  popup templates still do not.
* `{{PAGENAME}}` and relative links in popup templates refer to the page instead of `Badtitle/Parser`
* `<nowiki>`, `<pre>` and other tags in `{{#display_map:}}` marker text render instead of showing as strip markers
* Map wikitext counts toward the page's expansion limits, so a page with a very large map can now reach them
* On Special:Ask and Special:CargoQuery, map wikitext and GeoJSON pages render as an anonymous reader would see them

Compared before and after with a script, the output of 15 representative maps is otherwise the same: 14 on a page,
rendered by the legacy parser and by Parsoid, and one on Special:Ask. The Cargo format, which only reads GeoJSON pages,
is unchecked: Cargo was not installed.

`MappingService::newMapDataFromProcessingResult()` and `newMapDataFromParameters()`,
`CargoOutputBuilder::buildOutputFromCargoData()`, `QueryHandler::setTemplate()` and the `GeoJsonFetcher` constructor now
take the parser. Only `MapsFactory::newGeoJsonFetcher()` has a caller outside Maps, Semantic Result Formats, hence its
optional parser.

Considered, omitted:
* The `MediaWiki:Maps` page: site configuration, read once per request and documented to take effect at the next parse
* Icons, image layers and overlays, and KML files: Maps looks up file URLs, not page content
* A dependency on GeoJSON pages that do not exist: URL sources would then show up as wanted pages. Creating such a page
  still does not refresh the maps that name it
* Semantic Compound Queries and other callers that get whichever parser Semantic MediaWiki last held: Maps uses that
  parser, as it does for the tracking category since #949; before, it
  parsed on a copy of MediaWiki's main parser, which can be just as unrelated
* Semantic Result Formats' filtered format, which calls `newGeoJsonFetcher()` without a parser: passing one is a change
  in that extension

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
When the page's parser only preprocesses, as Parsoid has it do for parser functions, the extension tags in map
wikitext stay unprocessed and their strip markers hold the tags' wikitext. Map wikitext is then parsed from that
wikitext in HTML mode, so it goes through the tag hooks and the sanitizer instead of reaching the popup HTML as written.

Semantic MediaWiki can hand result formats a parser whose parse is over, and a caller could hand Cargo or
`MapsFactory::newGeoJsonFetcher()` one that never parsed. Maps now uses a given parser only while it is parsing, and
otherwise its own parser for an anonymous reader, decided in one place.

On Special:Ask and Special:CargoQuery, which nothing caches, Maps parses for the requesting user, as before this
branch, so wikis whose extensions refuse pages to anonymous readers keep showing those maps to whoever may see them.

The release notes now name what an administrator may notice.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The refusal tests now also show that the allowed part of the text renders, so they fail when nothing is expanded.
New tests cover links in inline labels and shape text, a refused GeoJSON page still being recorded on the page, and
the fetcher returning the page's current revision, which the GeoJSON editor saves against.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Older Cargo versions pass the display format a ParserOutput while rendering a page. The Cargo format now treats only a
missing parser as a special page, so such renders get the anonymous reader's parser rather than the requesting
user's.

The GeoJSON page fixture moves to PageCreator, the two "leaves the previous parse untouched" tests share one helper,
and MapsFactory lists its parser methods depth first.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A parser can be given options without an output, which MediaWiki 1.43 reports as null. The release notes now also
name behavior switches and say that GeoJSON pages follow the restrictions that Lua's mw.title:getContent() does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Parsing marker text and popups with recursiveTagParseFully() on the page's parser made core render every link the page
had collected so far again for each text, so maps whose markers link pages took time quadratic in the number of
markers: 5.8 s for 800 such markers, against 0.37 s before.

Each text now gets a parse of its own on a copy of the page's parser, with a copy of the page's parser options, so the
page's template callbacks and reader still apply and the options keep recording on the page's output. Core's
mergeInternalMetaDataFrom() and mergeTrackingMetaDataFrom() add what the parse recorded to the page's output: used
options, flags, templates, links, categories and extension data such as the pages a callback left out. The page's own
earlier values are merged back afterwards, because the merge would replace page properties and extension data the page
already has, such as the semantic data that Semantic MediaWiki keeps there. Module styles and modules go along too.
Strip markers in the text keep working: their HTML goes along to the copy, and outside HTML mode their wikitext is
parsed, as before.

Each text again has the expansion limits of its own parse, and behavior switches in it no longer affect the page.
The release notes move to 15.1.0, since public method signatures change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Merging the page's earlier output back after each text, to restore the page properties and extension data that the
merge replaces, cost time that grows with the page for every text: with 300 Semantic MediaWiki subobjects on the page,
1600 markers took 25 s, and on MediaWiki 1.44 and later core added every link of the page again. Maps now removes from
the output of a text what is the page's to decide before merging it: page properties, such as the default sort key
and those of behavior switches, and the extension data in which Semantic MediaWiki and PageImages collect data about
the page, such as its semantic data and image candidates. The 1600 markers now take 1.05 s, against 1.19 s on master.

The parse of a text also takes the head items, modules and config vars that Semantic MediaWiki holds for the page,
such as the data of a chart above the map, as it did on master with Semantic MediaWiki 7.0 and older. These now reach
the page along with the head items, config vars, modules and styles of the text itself. Core merges them on their own,
so the title, indicators, table of contents and index policy stay the page's.

A nowiki strip marker in the text is read once when handed over, so it counts once toward the page's unstrip limit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New tests cover `{{PAGENAME}}` and extension tags in marker text, an indicator in marker text staying off the page,
the page's cache expiry following a time shown in marker text, and the script a packed gallery in marker text needs.
The nowiki test now uses wikitext that only a nowiki strip marker keeps literal. Tests whose protection another test
already gives are gone, and the test classes name every class they guard and say why they need the database.

The release notes give each fix its own line, note the Parsoid exception for strip markers and the reference lists
that popups no longer get, and say when existing pages pick up the new links.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s over

A copy of the page's parser shares the state that extensions keep on it, and starting the parse of each text resets
that state. On MediaWiki 1.43 and 1.44, TemplateStyles' cache of the page's styles then pointed at strip markers of a
text, so the page's own styles went missing when a map using the same styles came first. Each map now parses its texts
on a parser from the parser factory, whose strip markers take names after those of the page.

Strip markers in a text go along as their own kind, including those inside the content of others, so a tag inside a
nowiki renders and nowiki content inside a tag stays protected. Tags that parse wikitext, such as the captions of a
gallery, leave link placeholders that only the page's parser can fill in. It now fills them in before their HTML goes
along, so a caption no longer shows another link of the text.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…changes

The strip marker pattern is a constant built as core builds its own, `MapPrinter::getParser()` is called
`getRenderingParser()` like its counterparts, the three link tests share one data provider, and the tests name the page
they render in one place.

The release notes add language links to what map wikitext now applies to the page, correct which references lose their
list, and note that semantic map popups that link pages render faster while popup templates that use Lua take longer.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
MediaWiki 1.46 replaced tests/phpunit/phpunit.php with a configuration that composer generates, so these jobs run
PHPUnit that way and skip the coverage run. The master job may fail without failing the workflow.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… texts

New tests show that data an extension collects about the page in extension data survives marker text that collects
its own, that the section edit link and index switches in marker text leave the page alone, and that strip markers of
the page keep their content when the parse of the marker text adds its own. The jobs for MediaWiki 1.46 and master
fetch the PHPUnit configuration template, which the tarballs leave out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ders()

For a strip marker whose HTML holds link placeholders, as the captions of a gallery leave, the page's parser now
renders the strip marker with recursiveTagParseFully(). Maps called replaceLinkHolders() instead, which MediaWiki 1.46
hard-deprecates and core master removes.

The parse of a text no longer replaces extension data of the page. Every entry without a merge strategy stays out, as
extensions such as Semantic MediaWiki, PageImages, Widgets and Kartographer collect data about the page there; sets
that carry a merge strategy combine with the page's. A ParserOutput hands over all of its extension data only in its
serialization. The flags of behavior switches and other page-level settings stay out too, as MediaWiki 1.46 merges
them, so __NOINDEX__ in a popup template no longer makes the page noindex there.

The test helpers delegate to the default fetch callbacks rather than the static methods that MediaWiki 1.47
deprecates, the shared-state test allows the deprecated parser property it needs, and the test user has a name of its
own. CI on MediaWiki 1.46 and master lets tests without a @Covers annotation pass, as tests/phpunit/phpunit.php did.
The release notes name what map texts no longer render or vary.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The test is about the link placeholders of the legacy parser, which MediaWiki master no longer uses by default.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Widgets extension replaces its markers with the HTML of the widgets when the parse of the page ends. Map texts
are parsed on their own, so the widgets of a text, and those the page expands in the arguments of a map, are now
filled in before the text goes into the map's data. Before, a popup showed the raw marker, and a widget the page
expanded could not be filled in at all, as its marker sat JSON-escaped in the map's data. Nothing of a widget is
merged into the page's output, so the page's own widgets keep theirs, and no widget HTML lands unescaped in the map's
data.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The notes lead with what readers see, give the {{PAGENAME}} fix a line of its own, state what changes under Parsoid
on MediaWiki 1.44 and later, and add the TemplateStyles fix that a parser of their own brings to map texts.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant