Repository navigation
syslog: fix message handling and add safe partition rebuilds - #471
Conversation
Remove native and jQuery UI tooltips from message rows and recommendation indicators while retaining accessible labels and click-open details. Use form_selectable_cell with a 15px width for every indicator cell and constrain matching headers and cell padding. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the tooltip-removal and fixed-width recommendation-column changes.
No open issues 🎉
What changed in this PR
Follow-up to #470 that removes passive hover tooltips from Syslog message rows and recommendation indicators while preserving accessible labels and click-open details.
Changes:
- Remove row and lightbulb titles and message-tooltip initialization.
- Fix recommendation columns at 15px across System Logs, Alert Logs, grouped summaries, and expanded details.
- Update regression tests and documentation for the revised behavior.
| File | Description |
|---|---|
| tests/Unit/RecommendationsTest.php | Assert titles are absent and accessible indicator labels remain. |
| tests/Fixtures/compact_workspace.php | Declare 15px recommendation cells in the browser fixture. |
| tests/browser/threshold_messages.cjs | Check declared widths for individual and grouped rows. |
| tests/browser/compact_workspace.cjs | Check rendered widths and tooltip-free, request-free hovers. |
| syslog.php | Render recommendation cells with the helper’s 15px width argument. |
| README.md | Document fixed-width indicators and click-only details. |
| js/functions.js | Remove message-row tooltip setup and title assignment. |
| includes/recommendations.php | Remove indicator titles while retaining accessible labels. |
| includes/functions.php | Stop emitting message-row titles and update the PHPDoc. |
| css/search.css | Constrain indicator widths and padding; remove obsolete tooltip styles. |
| CHANGELOG.md | Record the tooltip removal and column-width fix. |
Review effort: Balanced
Use a fixed 20px width for recommendation headers and all message-row variants. Retain the centered header alignment and verify it alongside desktop and mobile column widths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fetch group members on demand in bounded pages using the rendered search and date range. Keep relative ranges fixed during expansion, and avoid collecting every member ID or rendering hidden occurrences. Order Cancel before Save and show the Font Awesome refresh spinner only while filter callbacks remain pending. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Render Save as a submit button so Cacti themes apply primary-action styling, keeping Cancel first and non-submitting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add ui-state-active to Save while retaining its submit type and the Cancel-first button order. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep the dialog form separate from the main search form and route Enter and Save through one submit handler. Preserve validation, active button styling, and the existing CSRF-protected save callback without navigation or duplicate handlers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Normalize program names before reference lookup and rule processing using bounded batches of distinct names. Preserve interior punctuation and retry the incoming batch on database failures without changing archived program names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Preserve messages in bounded copies, verify values before removing originals, and retain interrupted copies for recovery. Report database size and maintenance requirements while keeping the legacy converter compatible. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep log tables within the results area while resizing, preserve message ellipsis and wrapping, and use accessible export glyphs on both tabs. Remove the unreleased conversion wrapper and its compatibility paths. Give both remaining utilities Cacti-style help, version output, and separated maintenance warnings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cover the grouped detail error paths and row rendering so the patch coverage gate passes. Let the database compute the digest lookup hash, run fixture excerpts from temporary includes instead of eval, intercept the browser fixture over https, and use a literal regression database. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
somethingwithproof
left a comment
There was a problem hiding this comment.
Two blockers in cli/upgrade_partitioning.php, both reproduced against MariaDB 11.8 in Docker (B1 with time_zone=America/New_York).
1. A legacy DATETIME table in a DST zone can be left empty with no supported recovery (cli/upgrade_partitioning.php:321-326, :492-519)
The DATETIME->TIMESTAMP path breaks on local times that don't exist because of the spring-forward gap. With a legacy DATETIME syslog table and the server in America/New_York, one row at 2026-03-08 02:30:00 causes:
- strict mode: the batch INSERT fails with
Incorrect datetime value sql_mode=''(Cacti strips strict modes): MariaDB shifts the value to03:00:00and the<=>verification fails
In both cases the original is already renamed to syslog_rebuild, so the live syslog is an empty or partial replacement and the log views show nothing. Re-running fails the same way, --discard-data is refused while syslog_rebuild exists, and the README has no manual recovery. Devices that report UTC or another zone's time through date-mysql will regularly produce gap-hour values.
Please add a pre-flight check before the RENAME that rejects DATETIME values that don't round-trip in the session zone, for example:
SELECT COUNT(*) FROM src
WHERE CONVERT_TZ(CONVERT_TZ(logtime, @@session.time_zone, '+00:00'), '+00:00', @@session.time_zone) <> logtimeor convert explicitly and verify against the same conversion. Please also document a rollback (drop the replacement, rename _rebuild back).
2. Writers that are still running create duplicate seq values and a state the tool can't finish (cli/upgrade_partitioning.php:126-130, :366-377, :509-519)
This needs the operator to skip the documented step of stopping writers, but nothing enforces it, and the poller runs syslog_process every cycle unless the plugin is disabled. The CREATE TABLE ... LIKE replacement restarts AUTO_INCREMENT at 1, so live inserts reuse seq values that copied rows also get, and the (seq, logtime) key accepts both. COUNT(old) != COUNT(new) then fails on every retry, and --discard-data is refused. GET_LOCK only excludes a second rebuild.
With 1,000,000 seeded rows and one INSERT every 5 ms, the replacement ended with 1,003,421 rows and 128 duplicated seq values. The rebuild failed the count check, and a re-run after stopping the writer failed the same way. No rows were lost.
Please refuse to start unless processing is stopped (check that syslog_enabled is off and/or take the register_process_start('syslog','master',...) slot), and set AUTO_INCREMENT right after CREATE TABLE LIKE rather than at the end.
With writers stopped, the rebuild preserved row counts and checksums on MariaDB and MySQL 8.0, and a re-run after kill -9 finished with an identical checksum.
|
Legacy datetime just needs to be dropped. No support. I worried about carrying the sequence over. Surprising AI did not catch that. But that was yesterday when Microsoft wrecked Copilot Agent mode again. |
Purge Syslog Devices called syslog_db_execute() from the utilities hook without ever connecting, so includes/database.php was never loaded and the plugin was disabled (#472). Connect first, report a failed connection, qualify the Syslog database, and use NOT EXISTS so a NULL host_id cannot silently prevent purging. Address the #471 review of cli/upgrade_partitioning.php: - Refuse to start while Syslog processing is enabled, and hold the syslog/master process slot for the duration. - Copy stored and removed messages without seq. The replacement's AUTO_INCREMENT is set above the original high-water mark before the first row, so sequence values are never reused and copied rows cannot collide with live inserts. - Verify each 10,000-row batch by count and MD5 checksum against its source range, then checkpoint it in syslog_rebuild_progress. A retry deletes unverified rows above the checkpoint and resumes. - Widen syslog_incoming in place; its seq is an archival identity. - Drop legacy DATETIME conversion: refuse such history before any change unless --discard-data is used, and document manual conversion and rollback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… add hourly partitioning (#473) * syslog: fix device purge crash and harden partition rebuilds Purge Syslog Devices called syslog_db_execute() from the utilities hook without ever connecting, so includes/database.php was never loaded and the plugin was disabled (#472). Connect first, report a failed connection, qualify the Syslog database, and use NOT EXISTS so a NULL host_id cannot silently prevent purging. Address the #471 review of cli/upgrade_partitioning.php: - Refuse to start while Syslog processing is enabled, and hold the syslog/master process slot for the duration. - Copy stored and removed messages without seq. The replacement's AUTO_INCREMENT is set above the original high-water mark before the first row, so sequence values are never reused and copied rows cannot collide with live inserts. - Verify each 10,000-row batch by count and MD5 checksum against its source range, then checkpoint it in syslog_rebuild_progress. A retry deletes unverified rows above the checkpoint and resumes. - Widen syslog_incoming in place; its seq is an archival identity. - Drop legacy DATETIME conversion: refuse such history before any change unless --discard-data is used, and document manual conversion and rollback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * syslog: add hourly/daily partitioning levels and prune future partitions Add a Partitioning Level of Daily (default) or Hourly for syslog and syslog_removed, matching MacTrack's partition-size workflow: - Settings: rename "Data Retention Settings" to "Data Retention and Partitioning" and add the Partitioning Level dropdown. The level is applied only by cli/upgrade_partitioning.php, or by its new --partition=hourly|daily option, which saves it once confirmed. Until then maintenance follows the tables' existing layout, and Settings and Status show the change as pending. - Install: replace the unused "Partitions per Day" advisor option with the level, and add $syslog_install_options['partitioning']. - Limits: refuse to save settings, install, or rebuild when retention, today, the pre-create window and dMaxValue need more partitions than the server allows (8192; 1024 before MariaDB 10.0.4 and MySQL 5.7), or when Hourly is combined with Indefinite retention. Cacti runs the config_settings hook before settings.php saves, on 1.2 and 1.3. - Hourly maintenance creates hYYYYMMDDHH partitions and keeps at least two days ahead, as MacTrack does, so dMaxValue splits stay instant. Recovery checks only the horizon in the steady state and binary-searches a gap. - Queries: get_syslog_messages() always bounds logtime at now, so empty future partitions are pruned whatever the filter says. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * syslog: check partition limits against the tables' current layout too Maintenance keeps the tables' existing partitioning level until cli/upgrade_partitioning.php rebuilds them, so a settings save or upgrade is also checked against every level the syslog and syslog_removed tables currently use. This refuses, for example, Indefinite retention while the tables are still hourly, even when Daily is selected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * tests: bind parameters in the hourly pruning checks Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * syslog: refresh Status cards through callbacks and fix Add card Cacti develop wraps every plain select in select2, which reports a change through jQuery's trigger. The card controller listens with a native addEventListener, so choosing a card under "Add card" did nothing. Bind the add select with jQuery in the Status adapter, append and bind the card, save the layout, and keep the select2 display in sync. Changing the Refresh interval assigned window.location, a full page load, and the Status tab also registered Cacti's page refresh. The interval is now saved per user through a CSRF-protected status_refresh callback, and a client timer refreshes the cards in place, through the same path as the Refresh button. The timer stops once the grid leaves the page. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * tests: use https fixture URLs and an explicit sort in the Status card fixture Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * syslog: copy partition rebuilds in 50,000-row batches A 10M-row sweep on a 2 GiB buffer pool found batches of 10,000 to 65,536 rows within measurement noise of each other, with 50,000 fastest. The batch size is now a parameter so the regression test can keep exercising multi-batch resume on a small table. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * syslog: address Copilot review of the rebuild, installer and Status timer - Rebuild locking: cli/upgrade_partitioning.php and syslog_process.php now share one database lock (syslog_processing_lock_name()) instead of the Cacti process registry, which on 1.2.x can treat another PHP script as stale and takes over expired slots. GET_LOCK never waits for or terminates the other side, and is released when its connection closes. A single lock also avoids old servers dropping a first GET_LOCK when a second is taken. - --discard-data with no incoming/outbox widening no longer issues "TABLE_NAME IN ()". - README rollback restores only tables that still have a <table>_rebuild original and leaves completed tables untouched. - Status auto-refresh: a late callback from a grid replaced by ajax navigation no longer resets the current grid's shared timer. - Installer limit check (syslog_partition_install_error()): a submitted Daily level wins over the saved one, and truncate ignores the old layouts of both tables, which it drops and recreates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * syslog: hold the processing lock on its own connection; repair and settings checks - syslog_process.php takes the processing lock through the new syslog_processing_lock_acquire(), on a separate connection to the same server as $syslog_cnn, opened from the details Cacti recorded for it (bypassing the cached session and persistent handles). MySQL before 5.7.5 keeps one named lock per session, so the GET_LOCK calls made by partition maintenance and alert delivery on the Syslog connection could otherwise release it mid-run and let a rebuild overlap. - Repair (automatic and cli/syslog_schema.php) recreates a missing syslog table from the current retention rather than the install-time days, and checks the plan against the server partition limit before any DDL. - The Syslog settings save is refused when the Syslog server's version or table layouts cannot be read, rather than validating against the Cacti server's version and no layouts. The installer check refuses likewise when the existing layouts cannot be read. - README: the DATETIME round-trip check counts values UNIX_TIMESTAMP() cannot convert (NULL-safe comparison). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * changelog: keep only security, issue and feature entries, in a fixed order Every release section now lists security, then issue, then feature entries. Numbered entries come first in ascending issue or pull number, then unnumbered entries alphabetically. bug: and compat: entries become issue:, and issue#N - / issue#N separators become issue#N:. A single trailing "(issue #N)" or "(#N)" reference moves into the prefix. dev:, docs: and test: entries are dropped. This PR's entries are tagged #473, and the rebuild entries now describe the shared processing lock and 50,000-row batches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * syslog: keep the idle processing lock connection open for the whole run The processor's lock connection is idle after GET_LOCK, so the server would close it, and release the lock, once wait_timeout elapsed during a long processing run. Raise its session wait_timeout to at least a day before taking the lock and skip processing if the server does not accept it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * syslog: keep workers and schema repair out of a partition rebuild - Transfer workers receive the master's lock session (--lock-owner) and exit without writing unless that session still holds the processing lock. The rebuild, once it holds the lock, refuses while live workers remain registered in Cacti's process table (stale rows of crashed workers are ignored). Registration precedes the worker's check, so every worker is either seen by the rebuild or sees the rebuild's lock. - syslog_process.php and cli/upgrade_partitioning.php connect without schema repair (SYSLOG_DEFER_SCHEMA_REPAIR); the processor repairs only after taking the processing lock. Recreating a missing table from any other caller first takes the lock on a separate connection, so a page load can never recreate a table a rebuild has renamed away. - Settings and Status report a pending partitioning level whenever either message table differs from the selected level, and Status shows both tables' levels when they differ. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: TheWitness <copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
#471 replaced cli/updatedb_convert_ms.php with cli/upgrade_partitioning.php --convert-ms but did not tombstone it, so installs upgraded from the #449 build keep the old script. It truncates the message tables and ALTERs logtime to TIMESTAMP(3) in place, which fails on legacy TO_DAYS(logtime) partitioned tables with "Constant, random or timezone-dependent expressions in (sub)partitioning function are not allowed". Add it to manifest.json tombstones so syslog_prune_files() removes it on upgrade, and point the 5.0 release notes at the replacement. Co-authored-by: TheWitness <copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>



Description
Remove native and jQuery UI hover tooltips from message rows and recommendation
lightbulbs. Keep the accessible lightbulb label and the click-open details flyout.
Fix the recommendation column at 20 pixels on System Logs, Alert Logs, grouped
summaries, and expanded details. Use
form_selectable_cell()with'20px'as itsthird argument, center the header, and prevent padding from widening the column.
Load grouped messages only when expanded, in replaceable 50-message pages rather
than rendering every occurrence as hidden HTML. Preserve the displayed search
filters and time range, including frozen relative-date bounds, through a bounded
server-owned query context.
Add
cli/upgrade_partitioning.phpto rebuild stored/removed partitions tothe configured engine, retention and future horizon, preserving messages by
default.
--convert-msalso converts the message pipeline on MariaDB.Rename originals, copy at most 10,000 rows per batch using the actual primary
key, verify counts and values, restore sequence high-water marks, and only
then drop originals. Interrupted copies retain their source tables for retry.
Remove the unreleased
updatedb_convert_ms.php; use--convert-mson theunified preserving command instead.
The rebuild command reports estimated database size and warns that the database
server needs at least that much additional free space, plus temporary/binlog
headroom, and that large transfers can take several hours. Document the
maintenance window, writer shutdown, timezone handling, recovery, and the
difference between schema repair and repartitioning.
Give both remaining utilities Cacti-style help: version/copyright banners,
lowercase usage, descriptions and aligned options, with separate maintenance
paragraphs for the rebuild command. Both support
--version,-vand-V.Use an accessible Font Awesome export glyph on both log tabs without changing
the export callback. Correct header-width measurement, constrain combined column
widths, and keep the message column flexible when resizing columns on either
side. Normalize oversized saved widths and refit on viewport changes. Preserve
Alert Logs message ellipsis and All Text wrapping; clip long header labels
without clipping resize handles. Mobile results keep their existing internal
scrolling behavior.
Trim leading and trailing parentheses and square brackets from incoming program
names before reference lookup and rule processing. Process distinct names in
bounded batches, preserve interior punctuation, and retain incoming messages for
retry if normalization fails. Existing archived program names are not rewritten.
Put Cancel before the submit-type, primary Save button in Save Search As.
Give the dialog its own form so Enter and Save use the same callback POST without
submitting the main search form or navigating away.
Use a Font Awesome refresh glyph that
spins only while filter callbacks are pending, including overlapping requests.
Related Issue
Follow-up to #470, requested after its merge.
Motivation and Context
Hover tooltips are distracting. Full messages and recommendations remain
available by clicking the message or row, without passive hover callbacks.
Previously, opening Grouped Messages eagerly fetched every member of each
displayed group, potentially exhausting PHP memory before any group was expanded.
Paging the members bounds PHP output and memory independently of group size.
The old precision converter discarded stored messages, and schema repair does
not rebuild existing partition layouts. The unified utility provides an
explicit maintenance operation without silently changing production schemas.
How Has This Been Tested?
clicking still opens full escaped message details and recommendations.
and mobile viewport widths, with the header centered.
initial detail fetch; expansion, caching, pagination, failure/retry, dynamic
message details, and 20px column alignment work.
SQLite checks for stored/removed member filters, date bounds, and paging.
record for timestamp ties, microseconds, and maximum unsigned sequence IDs.
executed normalization SQL, the reported program names, batch boundaries,
unchanged interior punctuation, and database-failure retry handling.
CSRF-protected callback, empty-name validation, and callback-only refresh
spinning, including concurrent callbacks and failure cleanup.
real CLI argument parsing, size/space/duration warnings,
failure handling, and UTC partition planning.
incoming copies, repeated sequence values across batch boundaries, maximum
unsigned high-water marks, zero IDs/timestamps, binary/NULL values, legacy
DATETIME and TO_DAYS layouts, non-UTC interpretation, Aria, interrupted
creation/copy recovery, value mismatches, outbox failures, and explicit discard.
Added this regression to the existing database CI step.
production System Logs and Alert Logs row rendering: resizing before/after
the message, extreme drags, stale oversized saved widths, narrower viewports,
ellipsis and All Text wrapping. Export retains its existing callback.
Screenshots (if appropriate):
Not applicable.
Types of changes
Checklist:
Not applicable: this change updates the plugin README and changelog.