Repository navigation
feature: Preserve legacy DATETIME history with --source-time-zone - #475
Merged
Merged
Conversation
Installs created before TIMESTAMP partitioning store logtime as DATETIME, often partitioned by RANGE (TO_DAYS(logtime)). Such a table cannot be widened in place: TO_DAYS() over a TIMESTAMP is rejected as a "timezone-dependent expression in (sub)partitioning function", and the rebuild utility refused to preserve DATETIME history at all. cli/upgrade_partitioning.php now takes --source-time-zone=<zone>. Before anything changes it requires the zone to be an IANA name known to PHP (Cacti) and loaded in the Syslog database's time zone tables. It also requires PHP and the database to agree on the zone's UTC offset at every transition spanning the stored history and now, and every DATETIME value to exist in the zone (no daylight-saving gap, within TIMESTAMP range). It reports repeated-hour rows and warns when Cacti uses a different zone. syslog and syslog_removed are then renamed and copied into TIMESTAMP, UNIX_TIMESTAMP-partitioned replacements with the session in the source zone, so every batch checksum compares the same wall-clock text. The incoming queue is converted in place in that zone, to keep its sequence identities, and verified afterwards. The zone is recorded until completion, and a retry with a different zone is refused. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Timezone validation can miss database-only DST transitions and allow originals to be dropped after incorrect conversions, while --convert-ms can silently leave legacy DATETIME(3) queues unconverted.
3 open issues
What changed in this PR
Adds --source-time-zone to the partition-upgrade utility so legacy DATETIME history, including TO_DAYS-partitioned tables, can be preserved without a manual conversion.
Changes:
- Adds source-zone validation, zone-aware copying, and retry-zone checks.
- Converts legacy incoming queues in place while preserving sequence identities.
- Adds unit and database regression coverage and documents the migration workflow.
| File | Description |
|---|---|
| tests/Unit/PartitionRebuildTest.php | Covers zone validation, retry handling, and incoming-queue conversion. |
| tests/regression/partition_rebuild_database_test.php | Adds database checks for legacy conversion, DST boundaries, retries, and preserved values. |
| README.md | Replaces the manual conversion recipe with the source-zone workflow and recovery guidance. |
| cli/upgrade_partitioning.php | Implements source-zone validation, conversion, retry tracking, and CLI help. |
| CHANGELOG.md | Records support for preserving legacy DATETIME history. |
Review effort: Balanced
bmfmancini
previously approved these changes
Nov 20, 2026
…sion - Probe PHP against the database at both ends of every stored day, and every 15 minutes on days PHP has a transition, in batched queries, so a transition only stale database tzdata has is refused instead of silently shifting values. Regression: a DB-only 2025 summer DST for America/Mexico_City. - Select the incoming queue and outbox for --convert-ms by exact column type, so a DATETIME(3) queue is converted to TIMESTAMP(3) in the source zone instead of being skipped, and require timestamp(3) before recording millisecond precision. The widen digest now includes the fractional seconds. - Warn about each Cacti zone (PHP or poller) that differs from the source zone, not only when both differ. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolves SonarCloud php:S2077 on the Mexico City transition-type lookup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… query OFFSET is a reserved word in MariaDB 10.6+, so CI's 11.8 rejected the unquoted column with a 1064 syntax error. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
bmfmancini
approved these changes
Nov 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.





Summary
Installs created before
TIMESTAMPpartitioning storelogtimeasDATETIME, often partitioned byRANGE (TO_DAYS(logtime)). Such a table cannot be widened in place. MariaDB rejectsTO_DAYS()over aTIMESTAMPwith:This is what a user on MariaDB 11.4.12 hit with the old
cli/updatedb_convert_ms.php(see #474). Until nowcli/upgrade_partitioning.phprefused to preserve DATETIME history, leaving--discard-dataor a manual conversion as the only options.This PR adds
--source-time-zone=<zone>:php cli/upgrade_partitioning.php --convert-ms --source-time-zone=America/New_YorkValidation before any change
These checks run while holding the processing lock, before the confirmation banner:
CONVERT_TZ()must resolve it, so the server's time zone tables must be loaded, as the Cacti installer already requires.mysql.time_zone*tables or stale PHP tzdata.DATETIMEmust round-trip throughUNIX_TIMESTAMP/FROM_UNIXTIMEin the zone. A value in a daylight-saving gap, or outside theTIMESTAMPrange, refuses the run and names a sample value, because it usually means the zone is wrong.Conversion
syslog/syslog_removed: the existing rename-and-copy rebuild, not an in-placeALTER. The replacement is created withTIMESTAMP/TIMESTAMP(3)andUNIX_TIMESTAMP(logtime)partitions. Rows are copied in 50,000-row batches with the session in the source zone. Each batch checksum comparesDATE_FORMAT(logtime)on both sides, which is the same wall-clock text, and the original is dropped only after the copy is verified.TIMESTAMPoriginals are still copied in UTC.syslog_incoming(with--convert-ms): converted in place in the source zone, so the sequence identities that archival relies on are kept. A row count and CRC of the wall-clock values are compared before and after.syslog_upart_source_time_zoneuntil the run completes. A retry that would resume an interrupted*_rebuildcopy with a different zone is refused.--source-time-zoneand--discard-data.Tests
PartitionRebuildTest): refused zone, zone passed through to the rebuild with the retry zone set and then cleared, mismatched resume zone refused, DATETIME incoming queue widened in the source zone. 20/20 pass; the full suite passes 856 tests.partition_rebuild_database_test.php, real MariaDB with tz tables): builds a DATETIME table partitioned byTO_DAYS, plus DATETIMEsyslog_removedandsyslog_incoming.Mars/Olympus, with a zone whose DST gap holds a stored value (Berlin), and with a gap value in New York. Each refusal leaves every table untouched.America/Chicagois refused, and a retry withAmerica/New_Yorkcompletes.syslog_incomingseq and instant,UNIX_TIMESTAMPpartition layout, and session zone restore are all checked.mariadb:11.8image loads the tz tables by default.cli/upgrade_partitioning.php.Docs
The README's legacy DATETIME section now documents the option and its checks, replacing the manual
ALTERrecipe. Help text and CHANGELOG are updated.Note: #474's release-notes line saying DATETIME tables "cannot be preserved automatically" should mention
--source-time-zoneonce both PRs are merged.