Repository navigation
syslog: fix device purge and Status cards, harden partition rebuilds, add hourly partitioning - #473
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved defects can let rebuilds overlap active processing, block --discard-data with invalid SQL, and cause the documented rollback to delete the only surviving copy of a completed table.
3 open issues
What changed in this PR
This PR addresses the Purge Syslog Devices crash reported in #472 and strengthens the partition rebuild utility introduced in #471.
Changes:
- Connect before purging, report connection failures, and use database-qualified, NULL-safe deletion queries.
- Add rebuild writer checks, sequence renumbering, batch verification and recovery checkpoints; preserve incoming identities through in-place conversion.
- Expand regression coverage and document maintenance, legacy DATETIME handling, and recovery.
| File | Description |
|---|---|
tests/Unit/PartitionRebuildTest.php |
Covers processing guards, DATETIME refusal, incoming conversion, and help output. |
tests/Security/CsrfPurgeTest.php |
Exercises real connection setup and failed-connection handling. |
tests/regression/partition_rebuild_database_test.php |
Tests batch verification, interrupted rebuild recovery, and identity preservation. |
README.md |
Documents rebuild behavior, maintenance requirements, and recovery procedures. |
locales/po/cacti.pot |
Adds the purge connection-failure message. |
includes/utilities.php |
Initializes the database connection and updates purge queries. |
cli/upgrade_partitioning.php |
Adds rebuild guards, checkpoints, verification, and in-place queue conversion. |
CHANGELOG.md |
Records the purge fix and partition rebuild hardening. |
Review effort: Balanced
|
More changes coming. This is draft. |
Purge Syslog Devices called syslog_db_execute() from the utilities hook without ever connecting, so includes/database.php was never loaded and the plugin was disabled (#472). Connect first, report a failed connection, qualify the Syslog database, and use NOT EXISTS so a NULL host_id cannot silently prevent purging. Address the #471 review of cli/upgrade_partitioning.php: - Refuse to start while Syslog processing is enabled, and hold the syslog/master process slot for the duration. - Copy stored and removed messages without seq. The replacement's AUTO_INCREMENT is set above the original high-water mark before the first row, so sequence values are never reused and copied rows cannot collide with live inserts. - Verify each 10,000-row batch by count and MD5 checksum against its source range, then checkpoint it in syslog_rebuild_progress. A retry deletes unverified rows above the checkpoint and resumes. - Widen syslog_incoming in place; its seq is an archival identity. - Drop legacy DATETIME conversion: refuse such history before any change unless --discard-data is used, and document manual conversion and rollback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a Partitioning Level of Daily (default) or Hourly for syslog and syslog_removed, matching MacTrack's partition-size workflow: - Settings: rename "Data Retention Settings" to "Data Retention and Partitioning" and add the Partitioning Level dropdown. The level is applied only by cli/upgrade_partitioning.php, or by its new --partition=hourly|daily option, which saves it once confirmed. Until then maintenance follows the tables' existing layout, and Settings and Status show the change as pending. - Install: replace the unused "Partitions per Day" advisor option with the level, and add $syslog_install_options['partitioning']. - Limits: refuse to save settings, install, or rebuild when retention, today, the pre-create window and dMaxValue need more partitions than the server allows (8192; 1024 before MariaDB 10.0.4 and MySQL 5.7), or when Hourly is combined with Indefinite retention. Cacti runs the config_settings hook before settings.php saves, on 1.2 and 1.3. - Hourly maintenance creates hYYYYMMDDHH partitions and keeps at least two days ahead, as MacTrack does, so dMaxValue splits stay instant. Recovery checks only the horizon in the steady state and binary-searches a gap. - Queries: get_syslog_messages() always bounds logtime at now, so empty future partitions are pruned whatever the filter says. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Maintenance keeps the tables' existing partitioning level until cli/upgrade_partitioning.php rebuilds them, so a settings save or upgrade is also checked against every level the syslog and syslog_removed tables currently use. This refuses, for example, Indefinite retention while the tables are still hourly, even when Daily is selected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
b80da8e to
69ca410
Compare
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cacti develop wraps every plain select in select2, which reports a change through jQuery's trigger. The card controller listens with a native addEventListener, so choosing a card under "Add card" did nothing. Bind the add select with jQuery in the Status adapter, append and bind the card, save the layout, and keep the select2 display in sync. Changing the Refresh interval assigned window.location, a full page load, and the Status tab also registered Cacti's page refresh. The interval is now saved per user through a CSRF-protected status_refresh callback, and a client timer refreshes the cards in place, through the same path as the Refresh button. The timer stops once the grid leaves the page. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… fixture Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A 10M-row sweep on a 2 GiB buffer pool found batches of 10,000 to 65,536 rows within measurement noise of each other, with 50,000 fastest. The batch size is now a parameter so the regression test can keep exercising multi-batch resume on a small table. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Partition rebuild: analysis summaryCopy design (review of #471). DATETIME → TIMESTAMP (10M rows, MariaDB 10.5, 2 GiB buffer pool):
Conversion costs almost nothing in speed; its risk is correctness (values in the DST gap fail or shift, and the repeated hour is ambiguous). Legacy DATETIME history is therefore refused before any change; Batch size (10M rows, 2 GiB buffer pool, full preserving rebuild each run):
The default is now 50,000 rows per batch. |
There was a problem hiding this comment.
🟡 Changes recommended
The rebuild CLI’s process-registration check does not enforce mutual exclusion across PHP entrypoints, allowing table rebuilds to overlap a still-running Syslog writer.
6 open issues
Discard fails when no tables need conversion Rebuild can overlap an active syslog process Ignore detached grid callbacks before resetting refresh timer · New Validate explicit partitioning level instead of saved setting · New Exclude replaced message tables from truncate validation · New Rollback can delete the only surviving table copy
Review effort: Balanced
…imer - Rebuild locking: cli/upgrade_partitioning.php and syslog_process.php now share one database lock (syslog_processing_lock_name()) instead of the Cacti process registry, which on 1.2.x can treat another PHP script as stale and takes over expired slots. GET_LOCK never waits for or terminates the other side, and is released when its connection closes. A single lock also avoids old servers dropping a first GET_LOCK when a second is taken. - --discard-data with no incoming/outbox widening no longer issues "TABLE_NAME IN ()". - README rollback restores only tables that still have a <table>_rebuild original and leaves completed tables untouched. - Status auto-refresh: a late callback from a grid replaced by ajax navigation no longer resets the current grid's shared timer. - Installer limit check (syslog_partition_install_error()): a submitted Daily level wins over the saved one, and truncate ignores the old layouts of both tables, which it drops and recreates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Critical defects can bypass retention safeguards when Syslog metadata is unavailable, allow processing to overlap rebuilds on older MySQL versions, and block missing-table repair through stale retention settings.
4 open issues
6 resolved since last review
Discard fails when no tables need conversion Rebuild can overlap an active syslog process Exclude replaced message tables from truncate validation Validate explicit partitioning level instead of saved setting Ignore detached grid callbacks before resetting refresh timer Rollback can delete the only surviving table copy
Review effort: Balanced
…ttings checks - syslog_process.php takes the processing lock through the new syslog_processing_lock_acquire(), on a separate connection to the same server as $syslog_cnn, opened from the details Cacti recorded for it (bypassing the cached session and persistent handles). MySQL before 5.7.5 keeps one named lock per session, so the GET_LOCK calls made by partition maintenance and alert delivery on the Syslog connection could otherwise release it mid-run and let a rebuild overlap. - Repair (automatic and cli/syslog_schema.php) recreates a missing syslog table from the current retention rather than the install-time days, and checks the plan against the server partition limit before any DDL. - The Syslog settings save is refused when the Syslog server's version or table layouts cannot be read, rather than validating against the Cacti server's version and no layouts. The installer check refuses likewise when the existing layouts cannot be read. - README: the DATETIME round-trip check counts values UNIX_TIMESTAMP() cannot convert (NULL-safe comparison). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…order Every release section now lists security, then issue, then feature entries. Numbered entries come first in ascending issue or pull number, then unnumbered entries alphabetically. bug: and compat: entries become issue:, and issue#N - / issue#N separators become issue#N:. A single trailing "(issue #N)" or "(#N)" reference moves into the prefix. dev:, docs: and test: entries are dropped. This PR's entries are tagged #473, and the rebuild entries now describe the shared processing lock and 50,000-row batches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
The idle lock connection can time out during ingestion, releasing the lock and allowing a rebuild to overlap processing because lock ownership is neither maintained nor monitored.
1 open issue
4 resolved since last review
Review effort: Balanced
The processor's lock connection is idle after GET_LOCK, so the server would close it, and release the lock, once wait_timeout elapsed during a long processing run. Raise its session wait_timeout to at least a day before taking the lock and skip processing if the server does not accept it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Transfer workers receive the master's lock session (--lock-owner) and exit without writing unless that session still holds the processing lock. The rebuild, once it holds the lock, refuses while live workers remain registered in Cacti's process table (stale rows of crashed workers are ignored). Registration precedes the worker's check, so every worker is either seen by the rebuild or sees the rebuild's lock. - syslog_process.php and cli/upgrade_partitioning.php connect without schema repair (SYSLOG_DEFER_SCHEMA_REPAIR); the processor repairs only after taking the processing lock. Recreating a missing table from any other caller first takes the lock on a separate connection, so a page load can never recreate a table a rebuild has renamed away. - Settings and Status report a pending partitioning level whenever either message table differs from the selected level, and Status shows both tables' levels when they differ. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
There was a problem hiding this comment.
🔵 Needs a closer look
Human assessment is needed to weigh unpublished findings about Never-refresh timer overflow and database lock failures being masked as ordinary contention.
No open issues 🎉
4 resolved since last review
Review effort: Balanced




Description
Purge Syslog Devices crash (#472). Cacti loads only
includes/utilities.phpfor theutilities_actionhook. The purge calledsyslog_db_execute()withoutsyslog_connect(),so
includes/database.phpwas never loaded, the request died withCall to undefined function syslog_db_execute(), and Cacti disabled the plugin. The purgenow connects first. A failed connection is logged and reported, and nothing is purged.
The deletes now name the configured Syslog database explicitly. They also use
NOT EXISTSinstead of
NOT IN (SELECT DISTINCT … UNION …): one NULLhost_idin the message tablesmade
NOT INpurge nothing, andNOT EXISTSprobes thehost_idindexes instead ofrunning
DISTINCTover every message.Partition rebuild hardening (review of #471).
cli/upgrade_partitioning.php:the syslog processing lock (
GET_LOCK(syslog_processing_lock_name(), 0)), whichsyslog_process.phpalso takes, and holds it for the whole rebuild. A poller cycletherefore cannot start processing partway through. Neither side waits on or terminates
the other: the rebuild refuses to start, and the processor logs a NOTE and skips the cycle.
The server releases the lock if the connection drops. The processor holds the lock on a
connection of its own (
syslog_processing_lock_acquire()). On MySQL before 5.7.5, asession keeps only one named lock, so the nested
GET_LOCKcalls that maintenance andalert delivery make on the Syslog connection could otherwise release it. That idle
connection's
wait_timeoutis raised to at least a day (and read back) so the servercannot drop it, and the lock, during a long run.
(
--lock-owner=<CONNECTION_ID>). A worker registers, then exits without writing unless thatsession still holds the processing lock, so workers that are delayed, standalone, or
orphaned by a killed master never write during a rebuild. Once the rebuild holds the lock, it
refuses while live
syslog/childprocesses remain registered (stale rows from crashedworkers are ignored). Every worker is therefore either seen by the rebuild or sees its lock.
syslog_process.phpand the rebuild connect without automatic repair(
SYSLOG_DEFER_SCHEMA_REPAIR); the processor repairs only once it holds the lock. Any othercaller, such as a page load, takes the lock on a separate connection before recreating a
missing table, and skips the repair while a run or rebuild holds it. A table the rebuild has
renamed away is never recreated underneath it. Repair sizes a recreated table from the
current retention and refuses, before any DDL, a plan above the partition limit.
seqinINSERT … SELECT. Stored and removed messages are copied withoutseq,in their original
(seq, logtime)order. This matches how the plugin already moves rowsbetween
syslog_incoming,syslogandsyslog_removed. The replacement'sAUTO_INCREMENTis set above the original high-water mark (GREATEST(MAX(seq) + 1, AUTO_INCREMENT)) before the first row is copied. Sequence values are therefore neverreused, and copied rows cannot collide with live inserts, which was the 128-duplicate
failure in the review. The command refuses to start if too few sequence values remain
above the high-water mark.
row count and two MD5-based aggregates covering every copied column. The comparison is
binary-exact, and
logtimeis normalized to%f. A matching batch is recorded insyslog_rebuild_progress. A retry deletes replacement rows above the last checkpoint andresumes after it. An empty replacement left by an interrupted setup is recreated. The
original is dropped only after every batch and the total counts match.
syslog_incomingis widened in place withALTER TABLE … MODIFYunder--convert-ms.Its
seqis an identity, stored asreplication_source_event_idand used to deduplicatearchival, so it is not renumbered. The outbox is widened in place, as before.
DATETIMEis not supported. It is detected before anything is renamed. Apreserving rebuild refuses with no changes;
--discard-datamay replace such a table.This removes the DST gap/overlap failure mode (review item 1). The README documents a
manual conversion that checks the values first, and how to roll back an abandoned rebuild.
Related Issue
Fixes #472. Addresses @somethingwithproof's review of #471.
Hourly and Daily partitioning
Partitioning Level dropdown (
syslog_partition_granularity): Daily (default) or Hourly.There is no Disabled level, because partitioned tables are the only supported architecture.
cli/upgrade_partitioning.php --partition=hourly|daily, which saves it once confirmed. Theutility rebuilds both tables to that level, preserving data.
Poller maintenance keeps following the size the tables actually have (
hYYYYMMDDHH/dYYYYMMDDnames). Settings and the Status tab show the change as pending until bothsyslogandsyslog_removeduse the selected level; if a rebuild finishes only one of them,Status shows each table's level. This part deliberately differs from MacTrack.
silent installs take
$syslog_install_options['partitioning'] = 'day'|'hour'.permitted partition counts. The required count is (retention + today + pre-create window) ×
24 for Hourly, +
dMaxValue. The limit is 8192, or 1024 on MariaDB < 10.0.4 and MySQL < 5.7.Hourly with Indefinite retention is also refused.
config_settingshook, before Cacti'ssettings.phpwritesanything, on both 1.2.x and 1.3.
could be given Indefinite retention by selecting Daily before the rebuild runs.
the current retention, and refuses before any DDL if that plan exceeds the limit.
never checked against fallback data.
MacTrack's
MACTRACK_PARTITION_AHEAD = 2 * 86400, or the configured window if larger. Newpartitions are therefore always split from an empty
dMaxValue, which is instant.get_syslog_messages()always addslogtime < FROM_UNIXTIME(now + 1),whatever the filter says. This covers both search modes, grouped and UNION views, export, and
group detail pages. Empty future partitions are therefore always pruned; this was verified
with
EXPLAIN PARTITIONSon MariaDB. Messages with future timestamps are hidden until thattime; that tradeoff was chosen deliberately.
Syslog Status page
selectin select2. Select2reports a choice with jQuery's
.trigger('change'), which never reaches the nativeaddEventListener('change')in the card controller (cards.js, the same code as core'sinclude/js/cards.js). The Status adapter now binds the add select with jQuery. Itappends and binds the chosen card at the end, saves the layout, and keeps the select2
display in sync. Core's
cards.jshas the same latent issue for any otheraddControluser.
window.location, and theStatus tab also registered Cacti's page refresh. The interval is now saved per user
through a CSRF-protected
status_refreshcallback. A client timer refreshes the cards inplace, through the same path as the Refresh button, and stops once the grid leaves the
page.
tests/browser/status_cards.cjsapplies select2 the way Cacti does. It fails ondevelopand passes here, with no page navigation.StatusRefreshCallbackTestcovers theendpoint's validation and CSRF checks.
Copy batch-size sweep (10M rows, 2 GiB buffer pool)
Each row is a full preserving rebuild of the same 10M-row, 4.8 GiB table on MariaDB 10.5
(96 MiB redo log, binlog off). Run-to-run noise is about ±7%: two 10,000 runs took 511 s
and 550 s.
were the same at every size.
syslog_upart_rebuild()parameter; the regression test passes 10,000 to keep multi-batchresume covered on a small table.
The boundary lookup is the indexed
ORDER BY seq, logtime LIMIT 1 OFFSET N-1, not a servercursor. Replacing it with fixed
seq BETWEENwindows computed in PHP removed the lookup,but the copy got slower by about the same amount:
The lookup is kept, because it also handles gaps in
seq; empty windows would otherwisefail verification.
Motivation and Context
seqvalues,when writers ran or DATETIME history crossed a DST change.
Impact assessment and timings
Setup. WSL2 (AlmaLinux 9), MariaDB 10.5.29, InnoDB, 2 GiB buffer pool (raised from
128 MiB for these runs), binlog off,
innodb_flush_log_at_trx_commit=1. Tables use theproduction
syslogschema with seven secondary indexes and 30 days of daily partitions.The 10M-row table is 4.9 GiB (data + indexes).
developtool, TIMESTAMP rebuild, preserve (seqcopied)seq), final verify join 43 s--convert-ms(TIMESTAMP→TIMESTAMP(3))--discard-datadeveloptool, batched DATETIME→TIMESTAMP copyFROM_UNIXTIME(UNIX_TIMESTAMP(logtime)) <> logtimeALTERDATETIME→TIMESTAMP + epoch repartitionits risk is correctness. A detect-and-convert-only-when-needed path would save about 3%.
developtool. Dropping theseqanti-join makes the copy statements 17% faster, which pays for most of the per-batch
verification and checkpointing that make a resume safe.
(TIMESTAMP);
syslog_removedis extra. About 55 minutes for the in-place DATETIMEALTER.Allow at least the reported database size in free space. The 100M-row confirmation run was
stopped at the maintainer's request, because the 10M results extrapolate linearly.
Partition granularity. This PR adds Hourly alongside Daily (see above). Any existing
RANGE layout can be rebuilt to either level. The MariaDB regression test rebuilds daily →
hourly → daily, preserving every row.
How Has This Been Tested?
with
tests/bin/patch-coverage.php. The one warning is the existing xdebug coverage notice inQueryBuilderGuardTest. New tests:PartitionGranularityTestcovers server limits by version, projected counts, thesettings-save gate (including hourly tables with Daily selected, and an unavailable Syslog
database), layout detection and lookup failures, and bounded hourly recovery where the
steady state reads the horizon once.
ProcessingLockTestcovers the processor's separate lock connection on Cacti 1.2 and 1.3connection details, the raised
wait_timeout(and refusal when the server keeps itshort), release, and the worker ownership check. It was also checked live against MariaDB
with both Cacti versions'
db_connect_real().cli_schema_upgrade_testchecks that a missing table is repaired only under theprocessing lock and is left alone while another session holds it.
--partitionCLI paths.review_fixes_database_testinstalls hourly, checks there are at least 2 days ahead, mergessix future hours into
dMaxValue, and confirms maintenance heals them while Settingsalready asks for Daily. It also checks that repair recreates a missing hourly table from the
current retention, and that it creates nothing for a refused plan.
partition_rebuild_database_test(71 checks) adds the daily → hourly → daily rebuilds, anEXPLAIN PARTITIONSproof of pruning, and a check that an hourly plan above the limit isrefused.
incoming_preservation_database_testhangs on this local MariaDB 10.5 even on unmodifieddevelop, so it was not used here.CsrfPurgeTestdrives the purge through the realsyslog_connect()andincludes/database.php, stubbing only Cacti'sdb_*layer. It fails ondevelopwith the exact Syslog 5 Develop version, Utilities -> Purge Syslog Devices, crash syslog plugin #472 fatal, and covers a Syslog database that cannot be reached.
PartitionRebuildTestcovers the refusal while Syslog is enabled, the processing lock heldelsewhere, releasing the lock after a refusal, live transfer workers (live, stale and
other-user pids), the DATETIME refusal before any change, and widening incoming in place.
partition_rebuild_database_testchecks:corruption are each detected or discarded, and the resume finishes.
seqvalues getdistinct new ones, and none is at or below the old high-water mark. Binary, zero-date
and NULL values survive.
another connection holds the processing lock.
--convert-mspipeline keeps incomingseqidentities and outbox instants, andrestores the session and lock after a failure.
check-i18n-pot.phppasses;locales/po/cacti.potwas regenerated with thebuild_gettext.shxgettext invocation (.po/.mofiles untouched).Types of changes
The rebuild now renumbers
seqin rebuilt stored/removed tables. No code reads old values:syslog_logs.logseqis written but never queried.Checklist:
Not applicable: README and CHANGELOG are updated in this PR. The CHANGELOG now keeps only
security,issueandfeatureentries, ordered by category, then ascending issue/PRnumber, then alphabetically;
bug:/compat:becameissue:anddev:/docs:/test:lines were dropped.