Repository navigation
feat(sources): read stored messages again with the rules - #100
Merged
Merged
Conversation
A rule applied only to what arrived after it was saved; the messages its device had sent before kept their old reading, and so did everything an older version had misread. Settings > Sources now reads stored messages again from their raw text, with the rules as they are now: from every device or from one rule's addresses, received in the last day, week, month or at any time. - Engine: RereadHistory starts a background pass over what was stored when it began, a batch at a time (MessagesUpTo), each message read by the source reader at its own arrival time (syslog.ReadAt), so a missing timestamp and an RFC 3164 year come out as they did. Only what changed is rewritten (RewriteReadings): time, levels, host, application, ids, text, structured data. The raw message, source, protocol and arrival never change, so reading again after a rule is changed or gone undoes it. Imported lines are left alone. - The search index follows the text: the old text out, the new in, in the same transaction, without changing the schema of any database. - GetHistoryReread, CancelHistoryReread and the history:reread event follow it; the history is shown again when it changed something. On Backend, so it runs where the history is: in the service when the window drives one. A window keeping the history of a sealing service reads its own database with the service's rules (RereadOptions.Rules). - Settings: a tab's content no longer shrinks to fit the body, which squeezed the rule list once the Sources tab grew; it scrolls instead. Tests: the index after a rewrite (FTS5 integrity check), and a pass end to end - one address, a time range, Windows-1252 decoded after the fact and found by search, imported lines untouched, a deleted rule undone, rules passed in.
This was referenced Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #99, whose address matching it uses for the "one rule's addresses" choice. It will be retargeted to
mainonce #99 is merged.A source rule applied only to messages that arrived after it was saved. The messages its device had sent before kept their old reading, and so did everything an older version had misread. An example: an rsyslog forward stored by v1.9 with
2026-10-27T09as its application.For the user
Settings › Sources › Messages already stored reads stored messages again from their raw text, with the rules as they are now.
How
RereadHistory): starts a background pass over the messages stored when it began, a batch at a time (LogStore.MessagesUpTo). New arrivals do not lengthen the pass.syslog.ReadAt,sources.Reader.ReadAt). A missing timestamp and the year of an RFC 3164 stamp therefore come out as they did on arrival.LogStore.RewriteReadings): time, levels, host, application, ids, text and structured data.GetHistoryReread,CancelHistoryRereadand thehistory:rereadevent report it.Backend, so the pass runs where the history is (in the service, when the window drives one). A window keeping the history of a sealing service reads its own database with the service's rules (RereadOptions.Rules, filled bysealedBackend).Tests
internal/storage: a rewrite keeps the FTS5 index in step. The new text is found, the old text is not, andintegrity-checkpasses.internal/engine, end to end:Checked:
go vet(Windows, Linux),staticcheck,go test -race ./...,svelte-check --fail-on-warnings,npm test,npm run build, the docs' links.