Skip to content

feat(sources): read stored messages again with the rules - #100

Merged
Wasabules merged 2 commits into
mainfrom
feat-reread-history
Oct 27, 2026
Merged

Wasabules merged 2 commits into
mainfrom
feat-reread-history

Conversation

@Wasabules

Copy link
Copy Markdown
Owner

Stacked on #99, whose address matching it uses for the "one rule's addresses" choice. It will be retargeted to main once #99 is merged.

A source rule applied only to messages that arrived after it was saved. The messages its device had sent before kept their old reading, and so did everything an older version had misread. An example: an rsyslog forward stored by v1.9 with 2026-10-27T09 as its application.

For the user

Settings › Sources › Messages already stored reads stored messages again from their raw text, with the rules as they are now.

  • Which messages: from every device, or from the addresses of one rule.
  • Which period: received in the last 24 hours, 7 days, 30 days, or at any time. The period is based on arrival time, because a device with a wrong clock is often the reason for writing a rule.
  • Progress: it runs in the background with a progress bar, and can be stopped; whatever was already rewritten stays rewritten. The history refreshes when it ends.
  • Undo: the raw message, the source and the arrival time never change. Reading again after a rule has been changed or deleted undoes what it did.
  • Left as they are: imported lines keep their file format's reading. Alerts and routing are not run again.

How

  • Engine (RereadHistory): starts a background pass over the messages stored when it began, a batch at a time (LogStore.MessagesUpTo). New arrivals do not lengthen the pass.
    • Each message is read by the source reader at its own arrival time (syslog.ReadAt, sources.Reader.ReadAt). A missing timestamp and the year of an RFC 3164 stamp therefore come out as they did on arrival.
    • Only messages read differently are rewritten (LogStore.RewriteReadings): time, levels, host, application, ids, text and structured data.
  • Search index: the FTS index is updated in the same transaction (old text out, new text in), as the insert and delete triggers do. No database's schema changes.
  • Progress: GetHistoryReread, CancelHistoryReread and the history:reread event report it.
  • Where it runs: the methods are on Backend, so the pass runs where the history is (in the service, when the window drives one). A window keeping the history of a sealing service reads its own database with the service's rules (RereadOptions.Rules, filled by sealedBackend).
  • Settings fix: a tab's content no longer shrinks to fit the dialog. This was squeezing the rule list once the Sources tab grew; the tab now scrolls.
  • Translations and docs: the 8 locales, the user guide (Messages already stored), the README and the documentation page.

Tests

  • internal/storage: a rewrite keeps the FTS5 index in step. The new text is found, the old text is not, and integrity-check passes.
  • internal/engine, end to end:
    • a period that excludes everything;
    • one address only;
    • Windows-1252 decoded after the fact, then found by search;
    • an imported line left untouched;
    • a deleted rule undone by reading again;
    • rules passed in (the sealed path);
    • an invalid address refused.
  • UI checked through the screenshot bridge, in both themes.

Checked: go vet (Windows, Linux), staticcheck, go test -race ./..., svelte-check --fail-on-warnings, npm test, npm run build, the docs' links.

A rule applied only to what arrived after it was saved; the messages its
device had sent before kept their old reading, and so did everything an
older version had misread. Settings > Sources now reads stored messages
again from their raw text, with the rules as they are now: from every
device or from one rule's addresses, received in the last day, week,
month or at any time.

- Engine: RereadHistory starts a background pass over what was stored
  when it began, a batch at a time (MessagesUpTo), each message read by
  the source reader at its own arrival time (syslog.ReadAt), so a
  missing timestamp and an RFC 3164 year come out as they did. Only
  what changed is rewritten (RewriteReadings): time, levels, host,
  application, ids, text, structured data. The raw message, source,
  protocol and arrival never change, so reading again after a rule is
  changed or gone undoes it. Imported lines are left alone.
- The search index follows the text: the old text out, the new in, in
  the same transaction, without changing the schema of any database.
- GetHistoryReread, CancelHistoryReread and the history:reread event
  follow it; the history is shown again when it changed something. On
  Backend, so it runs where the history is: in the service when the
  window drives one. A window keeping the history of a sealing service
  reads its own database with the service's rules (RereadOptions.Rules).
- Settings: a tab's content no longer shrinks to fit the body, which
  squeezed the rule list once the Sources tab grew; it scrolls instead.

Tests: the index after a rewrite (FTS5 integrity check), and a pass end
to end - one address, a time range, Windows-1252 decoded after the
fact and found by search, imported lines untouched, a deleted rule
undone, rules passed in.
@Wasabules
Wasabules changed the base branch from fix-source-address-match to main October 27, 2026 10:30
@Wasabules Wasabules closed this Oct 27, 2026
@Wasabules Wasabules reopened this Oct 27, 2026
@Wasabules
Wasabules merged commit 22e7261 into main Oct 27, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant