Skip to content

syslog: fix message handling and add safe partition rebuilds - #471

Merged
TheWitness merged 10 commits into
developfrom
fix/remove-message-hover
Nov 20, 2026
Merged

TheWitness merged 10 commits into
developfrom
fix/remove-message-hover

Conversation

@TheWitness

@TheWitness TheWitness commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Description

Remove native and jQuery UI hover tooltips from message rows and recommendation
lightbulbs. Keep the accessible lightbulb label and the click-open details flyout.

Fix the recommendation column at 20 pixels on System Logs, Alert Logs, grouped
summaries, and expanded details. Use form_selectable_cell() with '20px' as its
third argument, center the header, and prevent padding from widening the column.

Load grouped messages only when expanded, in replaceable 50-message pages rather
than rendering every occurrence as hidden HTML. Preserve the displayed search
filters and time range, including frozen relative-date bounds, through a bounded
server-owned query context.

Add cli/upgrade_partitioning.php to rebuild stored/removed partitions to
the configured engine, retention and future horizon, preserving messages by
default. --convert-ms also converts the message pipeline on MariaDB.
Rename originals, copy at most 10,000 rows per batch using the actual primary
key, verify counts and values, restore sequence high-water marks, and only
then drop originals. Interrupted copies retain their source tables for retry.
Remove the unreleased updatedb_convert_ms.php; use --convert-ms on the
unified preserving command instead.

The rebuild command reports estimated database size and warns that the database
server needs at least that much additional free space, plus temporary/binlog
headroom, and that large transfers can take several hours. Document the
maintenance window, writer shutdown, timezone handling, recovery, and the
difference between schema repair and repartitioning.
Give both remaining utilities Cacti-style help: version/copyright banners,
lowercase usage, descriptions and aligned options, with separate maintenance
paragraphs for the rebuild command. Both support --version, -v and -V.

Use an accessible Font Awesome export glyph on both log tabs without changing
the export callback. Correct header-width measurement, constrain combined column
widths, and keep the message column flexible when resizing columns on either
side. Normalize oversized saved widths and refit on viewport changes. Preserve
Alert Logs message ellipsis and All Text wrapping; clip long header labels
without clipping resize handles. Mobile results keep their existing internal
scrolling behavior.

Trim leading and trailing parentheses and square brackets from incoming program
names before reference lookup and rule processing. Process distinct names in
bounded batches, preserve interior punctuation, and retain incoming messages for
retry if normalization fails. Existing archived program names are not rewritten.

Put Cancel before the submit-type, primary Save button in Save Search As.
Give the dialog its own form so Enter and Save use the same callback POST without
submitting the main search form or navigating away.
Use a Font Awesome refresh glyph that
spins only while filter callbacks are pending, including overlapping requests.

Related Issue

Follow-up to #470, requested after its merge.

Motivation and Context

Hover tooltips are distracting. Full messages and recommendations remain
available by clicking the message or row, without passive hover callbacks.

Previously, opening Grouped Messages eagerly fetched every member of each
displayed group, potentially exhausting PHP memory before any group was expanded.
Paging the members bounds PHP output and memory independently of group size.

The old precision converter discarded stored messages, and schema repair does
not rebuild existing partition layouts. The unified utility provides an
explicit maintenance operation without silently changing production schemas.

How Has This Been Tested?

  • Focused Pest regressions: no row or lightbulb titles; accessible labels retained.
  • Workspace browser regression: hovers open no tooltip and make no requests;
    clicking still opens full escaped message details and recommendations.
  • Browser measurements: lightbulb header and cells are exactly 20px at desktop
    and mobile viewport widths, with the header centered.
  • Grouped-message browser regression: a million-occurrence summary performs no
    initial detail fetch; expansion, caching, pagination, failure/retry, dynamic
    message details, and 20px column alignment work.
  • Query/date/search Pest suites: 52 tests, 413 assertions, including executed
    SQLite checks for stored/removed member filters, date bounds, and paging.
  • MariaDB 11.8: the production representative-ID aggregate selects the correct
    record for timestamp ties, microseconds, and maximum unsigned sequence IDs.
  • Incoming/archival/replication Pest suites: 34 tests, 131 assertions, including
    executed normalization SQL, the reported program names, batch boundaries,
    unchanged interior punctuation, and database-failure retry handling.
  • Workspace browser regression: Cancel/Save ordering, Enter-to-save with a single
    CSRF-protected callback, empty-name validation, and callback-only refresh
    spinning, including concurrent callbacks and failure cleanup.
  • PHP/JavaScript syntax, plugin manifest, and whitespace checks.
  • Migration/schema/include-path Pest regressions, including
    real CLI argument parsing, size/space/duration warnings,
    failure handling, and UTC partition planning.
  • MariaDB 11.8 migration regression covering multi-batch stored and
    incoming copies, repeated sequence values across batch boundaries, maximum
    unsigned high-water marks, zero IDs/timestamps, binary/NULL values, legacy
    DATETIME and TO_DAYS layouts, non-UTC interpretation, Aria, interrupted
    creation/copy recovery, value mismatches, outbox failures, and explicit discard.
    Added this regression to the existing database CI step.
  • Existing schema CLI regression and translation-template syntax checks.
  • Browser regressions exercise actual Cacti resize/persistence callbacks with
    production System Logs and Alert Logs row rendering: resizing before/after
    the message, extreme drags, stale oversized saved widths, narrower viewports,
    ellipsis and All Text wrapping. Export retains its existing callback.
  • Both CLI help layouts and version aliases are checked through real entrypoints.

Screenshots (if appropriate):

Not applicable.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation repository accordingly.
    Not applicable: this change updates the plugin README and changelog.

Remove native and jQuery UI tooltips from message rows and recommendation indicators while retaining accessible labels and click-open details. Use form_selectable_cell with a 15px width for every indicator cell and constrain matching headers and cell padding.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 16:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the tooltip-removal and fixed-width recommendation-column changes.

No open issues 🎉

What changed in this PR

Follow-up to #470 that removes passive hover tooltips from Syslog message rows and recommendation indicators while preserving accessible labels and click-open details.

Changes:

  • Remove row and lightbulb titles and message-tooltip initialization.
  • Fix recommendation columns at 15px across System Logs, Alert Logs, grouped summaries, and expanded details.
  • Update regression tests and documentation for the revised behavior.
File Description
tests/​Unit/​RecommendationsTest.php Assert titles are absent and accessible indicator labels remain.
tests/​Fixtures/​compact_workspace.php Declare 15px recommendation cells in the browser fixture.
tests/​browser/​threshold_messages.cjs Check declared widths for individual and grouped rows.
tests/​browser/​compact_workspace.cjs Check rendered widths and tooltip-free, request-free hovers.
syslog.php Render recommendation cells with the helper’s 15px width argument.
README.md Document fixed-width indicators and click-only details.
js/​functions.js Remove message-row tooltip setup and title assignment.
includes/​recommendations.php Remove indicator titles while retaining accessible labels.
includes/​functions.php Stop emitting message-row titles and update the PHPDoc.
css/​search.css Constrain indicator widths and padding; remove obsolete tooltip styles.
CHANGELOG.md Record the tooltip removal and column-width fix.

Review effort: Balanced

TheWitness and others added 2 commits November 19, 2026 16:15
Use a fixed 20px width for recommendation headers and all message-row variants. Retain the centered header alignment and verify it alongside desktop and mobile column widths.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fetch group members on demand in bounded pages using the rendered search and date range. Keep relative ranges fixed during expansion, and avoid collecting every member ID or rendering hidden occurrences.

Order Cancel before Save and show the Font Awesome refresh spinner only while filter callbacks remain pending.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@TheWitness TheWitness changed the title syslog: remove hover tooltips and fix the indicator width syslog: remove hovers and bound grouped message loading Nov 19, 2026
Comment thread includes/functions.php Fixed
TheWitness and others added 4 commits November 19, 2026 17:07
Render Save as a submit button so Cacti themes apply primary-action styling, keeping Cancel first and non-submitting.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add ui-state-active to Save while retaining its submit type and the Cancel-first button order.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep the dialog form separate from the main search form and route Enter and Save through one submit handler. Preserve validation, active button styling, and the existing CSRF-protected save callback without navigation or duplicate handlers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Normalize program names before reference lookup and rule processing using bounded batches of distinct names. Preserve interior punctuation and retry the incoming batch on database failures without changing archived program names.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@TheWitness TheWitness changed the title syslog: remove hovers and bound grouped message loading syslog: fix grouped loading, program names, and search controls Nov 19, 2026
Preserve messages in bounded copies, verify values before removing originals, and retain interrupted copies for recovery. Report database size and maintenance requirements while keeping the legacy converter compatible.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@TheWitness TheWitness changed the title syslog: fix grouped loading, program names, and search controls syslog: fix message handling and add safe partition rebuilds Nov 19, 2026
Keep log tables within the results area while resizing, preserve message ellipsis and wrapping, and use accessible export glyphs on both tabs.

Remove the unreleased conversion wrapper and its compatibility paths. Give both remaining utilities Cacti-style help, version output, and separated maintenance warnings.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cover the grouped detail error paths and row rendering so the patch
coverage gate passes. Let the database compute the digest lookup hash,
run fixture excerpts from temporary includes instead of eval, intercept
the browser fixture over https, and use a literal regression database.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sonarqubecloud

Copy link
Copy Markdown

@somethingwithproof somethingwithproof left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two blockers in cli/upgrade_partitioning.php, both reproduced against MariaDB 11.8 in Docker (B1 with time_zone=America/New_York).

1. A legacy DATETIME table in a DST zone can be left empty with no supported recovery (cli/upgrade_partitioning.php:321-326, :492-519)

The DATETIME->TIMESTAMP path breaks on local times that don't exist because of the spring-forward gap. With a legacy DATETIME syslog table and the server in America/New_York, one row at 2026-03-08 02:30:00 causes:

  • strict mode: the batch INSERT fails with Incorrect datetime value
  • sql_mode='' (Cacti strips strict modes): MariaDB shifts the value to 03:00:00 and the <=> verification fails

In both cases the original is already renamed to syslog_rebuild, so the live syslog is an empty or partial replacement and the log views show nothing. Re-running fails the same way, --discard-data is refused while syslog_rebuild exists, and the README has no manual recovery. Devices that report UTC or another zone's time through date-mysql will regularly produce gap-hour values.

Please add a pre-flight check before the RENAME that rejects DATETIME values that don't round-trip in the session zone, for example:

SELECT COUNT(*) FROM src
WHERE CONVERT_TZ(CONVERT_TZ(logtime, @@session.time_zone, '+00:00'), '+00:00', @@session.time_zone) <> logtime

or convert explicitly and verify against the same conversion. Please also document a rollback (drop the replacement, rename _rebuild back).

2. Writers that are still running create duplicate seq values and a state the tool can't finish (cli/upgrade_partitioning.php:126-130, :366-377, :509-519)

This needs the operator to skip the documented step of stopping writers, but nothing enforces it, and the poller runs syslog_process every cycle unless the plugin is disabled. The CREATE TABLE ... LIKE replacement restarts AUTO_INCREMENT at 1, so live inserts reuse seq values that copied rows also get, and the (seq, logtime) key accepts both. COUNT(old) != COUNT(new) then fails on every retry, and --discard-data is refused. GET_LOCK only excludes a second rebuild.

With 1,000,000 seeded rows and one INSERT every 5 ms, the replacement ended with 1,003,421 rows and 128 duplicated seq values. The rebuild failed the count check, and a re-run after stopping the writer failed the same way. No rows were lost.

Please refuse to start unless processing is stopped (check that syslog_enabled is off and/or take the register_process_start('syslog','master',...) slot), and set AUTO_INCREMENT right after CREATE TABLE LIKE rather than at the end.

With writers stopped, the rebuild preserved row counts and checksums on MariaDB and MySQL 8.0, and a re-run after kill -9 finished with an identical checksum.

@TheWitness
TheWitness merged commit 5b1bebe into develop Nov 20, 2026
11 checks passed
@TheWitness
TheWitness deleted the fix/remove-message-hover branch November 20, 2026 02:42
@TheWitness

Copy link
Copy Markdown
Member Author

Legacy datetime just needs to be dropped. No support. I worried about carrying the sequence over. Surprising AI did not catch that. But that was yesterday when Microsoft wrecked Copilot Agent mode again.

TheWitness pushed a commit that referenced this pull request Nov 20, 2026
Purge Syslog Devices called syslog_db_execute() from the utilities hook
without ever connecting, so includes/database.php was never loaded and the
plugin was disabled (#472). Connect first, report a failed connection,
qualify the Syslog database, and use NOT EXISTS so a NULL host_id cannot
silently prevent purging.

Address the #471 review of cli/upgrade_partitioning.php:

- Refuse to start while Syslog processing is enabled, and hold the
  syslog/master process slot for the duration.
- Copy stored and removed messages without seq. The replacement's
  AUTO_INCREMENT is set above the original high-water mark before the
  first row, so sequence values are never reused and copied rows cannot
  collide with live inserts.
- Verify each 10,000-row batch by count and MD5 checksum against its
  source range, then checkpoint it in syslog_rebuild_progress. A retry
  deletes unverified rows above the checkpoint and resumes.
- Widen syslog_incoming in place; its seq is an archival identity.
- Drop legacy DATETIME conversion: refuse such history before any
  change unless --discard-data is used, and document manual conversion
  and rollback.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TheWitness added a commit that referenced this pull request Nov 20, 2026
… add hourly partitioning (#473)

* syslog: fix device purge crash and harden partition rebuilds

Purge Syslog Devices called syslog_db_execute() from the utilities hook
without ever connecting, so includes/database.php was never loaded and the
plugin was disabled (#472). Connect first, report a failed connection,
qualify the Syslog database, and use NOT EXISTS so a NULL host_id cannot
silently prevent purging.

Address the #471 review of cli/upgrade_partitioning.php:

- Refuse to start while Syslog processing is enabled, and hold the
  syslog/master process slot for the duration.
- Copy stored and removed messages without seq. The replacement's
  AUTO_INCREMENT is set above the original high-water mark before the
  first row, so sequence values are never reused and copied rows cannot
  collide with live inserts.
- Verify each 10,000-row batch by count and MD5 checksum against its
  source range, then checkpoint it in syslog_rebuild_progress. A retry
  deletes unverified rows above the checkpoint and resumes.
- Widen syslog_incoming in place; its seq is an archival identity.
- Drop legacy DATETIME conversion: refuse such history before any
  change unless --discard-data is used, and document manual conversion
  and rollback.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* syslog: add hourly/daily partitioning levels and prune future partitions

Add a Partitioning Level of Daily (default) or Hourly for syslog and
syslog_removed, matching MacTrack's partition-size workflow:

- Settings: rename "Data Retention Settings" to "Data Retention and
  Partitioning" and add the Partitioning Level dropdown. The level is
  applied only by cli/upgrade_partitioning.php, or by its new
  --partition=hourly|daily option, which saves it once confirmed. Until
  then maintenance follows the tables' existing layout, and Settings and
  Status show the change as pending.
- Install: replace the unused "Partitions per Day" advisor option with the
  level, and add $syslog_install_options['partitioning'].
- Limits: refuse to save settings, install, or rebuild when retention,
  today, the pre-create window and dMaxValue need more partitions than
  the server allows (8192; 1024 before MariaDB 10.0.4 and MySQL 5.7), or
  when Hourly is combined with Indefinite retention. Cacti runs the
  config_settings hook before settings.php saves, on 1.2 and 1.3.
- Hourly maintenance creates hYYYYMMDDHH partitions and keeps at least two
  days ahead, as MacTrack does, so dMaxValue splits stay instant. Recovery
  checks only the horizon in the steady state and binary-searches a gap.
- Queries: get_syslog_messages() always bounds logtime at now, so empty
  future partitions are pruned whatever the filter says.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* syslog: check partition limits against the tables' current layout too

Maintenance keeps the tables' existing partitioning level until
cli/upgrade_partitioning.php rebuilds them, so a settings save or upgrade
is also checked against every level the syslog and syslog_removed tables
currently use. This refuses, for example, Indefinite retention while
the tables are still hourly, even when Daily is selected.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* tests: bind parameters in the hourly pruning checks

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* syslog: refresh Status cards through callbacks and fix Add card

Cacti develop wraps every plain select in select2, which reports a change
through jQuery's trigger. The card controller listens with a native
addEventListener, so choosing a card under "Add card" did nothing. Bind
the add select with jQuery in the Status adapter, append and bind the
card, save the layout, and keep the select2 display in sync.

Changing the Refresh interval assigned window.location, a full page
load, and the Status tab also registered Cacti's page refresh. The
interval is now saved per user through a CSRF-protected status_refresh
callback, and a client timer refreshes the cards in place, through the
same path as the Refresh button. The timer stops once the grid leaves
the page.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* tests: use https fixture URLs and an explicit sort in the Status card fixture

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* syslog: copy partition rebuilds in 50,000-row batches

A 10M-row sweep on a 2 GiB buffer pool found batches of 10,000 to 65,536
rows within measurement noise of each other, with 50,000 fastest. The
batch size is now a parameter so the regression test can keep exercising
multi-batch resume on a small table.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* syslog: address Copilot review of the rebuild, installer and Status timer

- Rebuild locking: cli/upgrade_partitioning.php and syslog_process.php now
  share one database lock (syslog_processing_lock_name()) instead of the
  Cacti process registry, which on 1.2.x can treat another PHP script as
  stale and takes over expired slots. GET_LOCK never waits for or
  terminates the other side, and is released when its connection closes.
  A single lock also avoids old servers dropping a first GET_LOCK when a
  second is taken.
- --discard-data with no incoming/outbox widening no longer issues
  "TABLE_NAME IN ()".
- README rollback restores only tables that still have a <table>_rebuild
  original and leaves completed tables untouched.
- Status auto-refresh: a late callback from a grid replaced by ajax
  navigation no longer resets the current grid's shared timer.
- Installer limit check (syslog_partition_install_error()): a submitted
  Daily level wins over the saved one, and truncate ignores the old
  layouts of both tables, which it drops and recreates.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* syslog: hold the processing lock on its own connection; repair and settings checks

- syslog_process.php takes the processing lock through the new
  syslog_processing_lock_acquire(), on a separate connection to the same
  server as $syslog_cnn, opened from the details Cacti recorded for it
  (bypassing the cached session and persistent handles). MySQL before
  5.7.5 keeps one named lock per session, so the GET_LOCK calls made by
  partition maintenance and alert delivery on the Syslog connection
  could otherwise release it mid-run and let a rebuild overlap.
- Repair (automatic and cli/syslog_schema.php) recreates a missing
  syslog table from the current retention rather than the install-time
  days, and checks the plan against the server partition limit before
  any DDL.
- The Syslog settings save is refused when the Syslog server's version
  or table layouts cannot be read, rather than validating against the
  Cacti server's version and no layouts. The installer check refuses
  likewise when the existing layouts cannot be read.
- README: the DATETIME round-trip check counts values UNIX_TIMESTAMP()
  cannot convert (NULL-safe comparison).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* changelog: keep only security, issue and feature entries, in a fixed order

Every release section now lists security, then issue, then feature
entries. Numbered entries come first in ascending issue or pull number,
then unnumbered entries alphabetically. bug: and compat: entries become
issue:, and issue#N - / issue#N separators become issue#N:. A single
trailing "(issue #N)" or "(#N)" reference moves into the prefix. dev:,
docs: and test: entries are dropped. This PR's entries are tagged #473,
and the rebuild entries now describe the shared processing lock and
50,000-row batches.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* syslog: keep the idle processing lock connection open for the whole run

The processor's lock connection is idle after GET_LOCK, so the server
would close it, and release the lock, once wait_timeout elapsed during a
long processing run. Raise its session wait_timeout to at least a day
before taking the lock and skip processing if the server does not
accept it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* syslog: keep workers and schema repair out of a partition rebuild

- Transfer workers receive the master's lock session (--lock-owner) and
  exit without writing unless that session still holds the processing
  lock. The rebuild, once it holds the lock, refuses while live workers
  remain registered in Cacti's process table (stale rows of crashed
  workers are ignored). Registration precedes the worker's check, so
  every worker is either seen by the rebuild or sees the rebuild's lock.
- syslog_process.php and cli/upgrade_partitioning.php connect without
  schema repair (SYSLOG_DEFER_SCHEMA_REPAIR); the processor repairs only
  after taking the processing lock. Recreating a missing table from any
  other caller first takes the lock on a separate connection, so a page
  load can never recreate a table a rebuild has renamed away.
- Settings and Status report a pending partitioning level whenever
  either message table differs from the selected level, and Status
  shows both tables' levels when they differ.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: TheWitness <copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TheWitness added a commit that referenced this pull request Nov 20, 2026
#471 replaced cli/updatedb_convert_ms.php with
cli/upgrade_partitioning.php --convert-ms but did not tombstone it, so
installs upgraded from the #449 build keep the old script. It truncates
the message tables and ALTERs logtime to TIMESTAMP(3) in place, which
fails on legacy TO_DAYS(logtime) partitioned tables with "Constant,
random or timezone-dependent expressions in (sub)partitioning function
are not allowed".

Add it to manifest.json tombstones so syslog_prune_files() removes it on
upgrade, and point the 5.0 release notes at the replacement.

Co-authored-by: TheWitness <copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants