Should I use a custom Lexer or custom Processor to highlight text in a document?

[sudomain]

Hello,

I'm writing a prompt_toolkit program that highlights text in a document based on quotes (strings) from the document that have been saved to a separate file (quotes.txt). It is a full-screen program based on the pager.py example.
I've been trying to implement a custom Lexer that does this, with limited success. I just found out about Processors and now I'm wondering if I should be using a custom Processor or maybe a different class?

Also, is it possible for a Lexer or Processor to be "refreshed" so that if I manually add strings that I want highlighted to quotes.txt, then prompt_toolkit can begin highlighting the new strings?

[nitanmarcel]

Noth should work almost the same I suppose. Processor seem to work on text input, lexing on content creation, which if I'm not wrong, should be called each time the app updates. So if you go with the lexer, app.invalidate should, If I'm not wrong re-lex the document

[nitanmarcel]

I've implemented an async lexer in my editor and I can confirm that invalidating the app with get_app().invalidate() does "manually" applies the lexer. So this is the way to go if you want a "refreshed" lexer

[tayfuryldz]

For this kind of highlighting I would use a Processor rather than a Lexer.

A Lexer answers "what syntax/style fragments does this document line contain?" and is a good fit when styling is derived from the text/language itself. Your quote set is external, mutable UI state layered on top of an otherwise normal document. Processors are designed for exactly that kind of transformation of the fragments produced for display; prompt_toolkit already uses processors for selections and incremental-search highlighting.

That also lets you keep whatever lexer you want for the document and add quote highlighting as an input processor on the BufferControl/TextArea.

When quotes.txt changes, update the in-memory set used by your processor and call get_app().invalidate(). The next render invokes the processor again with the current transformation state, so you do not need to replace/recreate the Buffer or Lexer just to refresh the highlighting.

One implementation detail matters: a Processor transforms fragments and must preserve a correct source-to-display cursor mapping. If you only change style strings while keeping the text and fragment lengths identical, that mapping is straightforward. Avoid inserting/removing display characters merely to highlight a match.

I would therefore parse/watch quotes.txt outside the rendering callback, keep a normalized collection of ranges/strings in memory, have the Processor apply styles to matching ranges, and invalidate when that collection changes. That separates file I/O/state updates from the rendering hot path.