Skip to content

feature: Preserve legacy DATETIME history with --source-time-zone - #475

Merged
TheWitness merged 4 commits into
developfrom
feature/source-time-zone
Nov 20, 2026
Merged

TheWitness merged 4 commits into
developfrom
feature/source-time-zone

Conversation

@TheWitness

Copy link
Copy Markdown
Member

Summary

Installs created before TIMESTAMP partitioning store logtime as DATETIME, often partitioned by RANGE (TO_DAYS(logtime)). Such a table cannot be widened in place. MariaDB rejects TO_DAYS() over a TIMESTAMP with:

Constant, random or timezone-dependent expressions in (sub)partitioning function are not allowed

This is what a user on MariaDB 11.4.12 hit with the old cli/updatedb_convert_ms.php (see #474). Until now cli/upgrade_partitioning.php refused to preserve DATETIME history, leaving --discard-data or a manual conversion as the only options.

This PR adds --source-time-zone=<zone>:

php cli/upgrade_partitioning.php --convert-ms --source-time-zone=America/New_York

Validation before any change

These checks run while holding the processing lock, before the confirmation banner:

  1. Known to Cacti: the zone must be an IANA name in PHP's time zone database, which is what Cacti uses.
  2. Known to the Syslog database: CONVERT_TZ() must resolve it, so the server's time zone tables must be loaded, as the Cacti installer already requires.
  3. Accurate: PHP and the database must give the same UTC offset for the zone just before and at every transition spanning the stored history, and at the current time. This catches stale mysql.time_zone* tables or stale PHP tzdata.
  4. Every value exists: each non-zero DATETIME must round-trip through UNIX_TIMESTAMP/FROM_UNIXTIME in the zone. A value in a daylight-saving gap, or outside the TIMESTAMP range, refuses the run and names a sample value, because it usually means the zone is wrong.
  5. Report: prints the number of rows in repeated fall-back hours (MariaDB uses the first occurrence) and warns when Cacti's PHP or poller zone differs from the one given.

Conversion

  • syslog / syslog_removed: the existing rename-and-copy rebuild, not an in-place ALTER. The replacement is created with TIMESTAMP/TIMESTAMP(3) and UNIX_TIMESTAMP(logtime) partitions. Rows are copied in 50,000-row batches with the session in the source zone. Each batch checksum compares DATE_FORMAT(logtime) on both sides, which is the same wall-clock text, and the original is dropped only after the copy is verified. TIMESTAMP originals are still copied in UTC.
  • syslog_incoming (with --convert-ms): converted in place in the source zone, so the sequence identities that archival relies on are kept. A row count and CRC of the wall-clock values are compared before and after.
  • Retries: the zone is stored in syslog_upart_source_time_zone until the run completes. A retry that would resume an interrupted *_rebuild copy with a different zone is refused.
  • Without the option, legacy tables are still refused, and the error now names both --source-time-zone and --discard-data.

Tests

  • Unit (PartitionRebuildTest): refused zone, zone passed through to the rebuild with the retry zone set and then cleared, mismatched resume zone refused, DATETIME incoming queue widened in the source zone. 20/20 pass; the full suite passes 856 tests.
  • Database regression (partition_rebuild_database_test.php, real MariaDB with tz tables): builds a DATETIME table partitioned by TO_DAYS, plus DATETIME syslog_removed and syslog_incoming.
    • Refused with no zone, with Mars/Olympus, with a zone whose DST gap holds a stored value (Berlin), and with a gap value in New York. Each refusal leaves every table untouched.
    • An injected copy failure keeps the original and the zone. A retry with America/Chicago is refused, and a retry with America/New_York completes.
    • Exact New York instants on both sides of the gap and in winter, repeated-hour handling, zero dates, syslog_incoming seq and instant, UNIX_TIMESTAMP partition layout, and session zone restore are all checked.
    • 100 checks pass. CI's mariadb:11.8 image loads the tz tables by default.
  • PHPStan level 6 is clean on cli/upgrade_partitioning.php.

Docs

The README's legacy DATETIME section now documents the option and its checks, replacing the manual ALTER recipe. Help text and CHANGELOG are updated.

Note: #474's release-notes line saying DATETIME tables "cannot be preserved automatically" should mention --source-time-zone once both PRs are merged.

Installs created before TIMESTAMP partitioning store logtime as DATETIME,
often partitioned by RANGE (TO_DAYS(logtime)). Such a table cannot be
widened in place: TO_DAYS() over a TIMESTAMP is rejected as a
"timezone-dependent expression in (sub)partitioning function", and the
rebuild utility refused to preserve DATETIME history at all.

cli/upgrade_partitioning.php now takes --source-time-zone=<zone>. Before
anything changes it requires the zone to be an IANA name known to PHP
(Cacti) and loaded in the Syslog database's time zone tables. It also
requires PHP and the database to agree on the zone's UTC offset at every
transition spanning the stored history and now, and every DATETIME value
to exist in the zone (no daylight-saving gap, within TIMESTAMP range).
It reports repeated-hour rows and warns when Cacti uses a different zone.

syslog and syslog_removed are then renamed and copied into TIMESTAMP,
UNIX_TIMESTAMP-partitioned replacements with the session in the source
zone, so every batch checksum compares the same wall-clock text. The
incoming queue is converted in place in that zone, to keep its sequence
identities, and verified afterwards. The zone is recorded until
completion, and a retry with a different zone is refused.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 20, 2026 20:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Timezone validation can miss database-only DST transitions and allow originals to be dropped after incorrect conversions, while --convert-ms can silently leave legacy DATETIME(3) queues unconverted.

3 open issues
What changed in this PR

Adds --source-time-zone to the partition-upgrade utility so legacy DATETIME history, including TO_DAYS-partitioned tables, can be preserved without a manual conversion.

Changes:

  • Adds source-zone validation, zone-aware copying, and retry-zone checks.
  • Converts legacy incoming queues in place while preserving sequence identities.
  • Adds unit and database regression coverage and documents the migration workflow.
File Description
tests/​Unit/​PartitionRebuildTest.php Covers zone validation, retry handling, and incoming-queue conversion.
tests/​regression/​partition_rebuild_database_test.php Adds database checks for legacy conversion, DST boundaries, retries, and preserved values.
README.md Replaces the manual conversion recipe with the source-zone workflow and recovery guidance.
cli/​upgrade_partitioning.php Implements source-zone validation, conversion, retry tracking, and CLI help.
CHANGELOG.md Records support for preserving legacy DATETIME history.

Review effort: Balanced

Comment thread cli/upgrade_partitioning.php Resolved
Comment thread cli/upgrade_partitioning.php Resolved
Comment thread cli/upgrade_partitioning.php Resolved
bmfmancini
bmfmancini previously approved these changes Nov 20, 2026
…sion

- Probe PHP against the database at both ends of every stored day, and
  every 15 minutes on days PHP has a transition, in batched queries, so
  a transition only stale database tzdata has is refused instead of
  silently shifting values. Regression: a DB-only 2025 summer DST for
  America/Mexico_City.
- Select the incoming queue and outbox for --convert-ms by exact column
  type, so a DATETIME(3) queue is converted to TIMESTAMP(3) in the source
  zone instead of being skipped, and require timestamp(3) before
  recording millisecond precision. The widen digest now includes the
  fractional seconds.
- Warn about each Cacti zone (PHP or poller) that differs from the
  source zone, not only when both differ.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TheWitness and others added 2 commits November 20, 2026 16:13
Resolves SonarCloud php:S2077 on the Mexico City transition-type lookup.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… query

OFFSET is a reserved word in MariaDB 10.6+, so CI's 11.8 rejected the
unquoted column with a 1064 syntax error.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sonarqubecloud

Copy link
Copy Markdown

@TheWitness
TheWitness merged commit 0a853df into develop Nov 20, 2026
8 checks passed
@TheWitness
TheWitness deleted the feature/source-time-zone branch November 20, 2026 22:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants