diff --git a/CHANGELOG.md b/CHANGELOG.md index e96db578..059e7379 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ --- develop --- +* feature: Preserve legacy DATETIME logtime history (including TO_DAYS partitioned tables) in cli/upgrade_partitioning.php with --source-time-zone=. Before any change the zone must be known to PHP/Cacti and the Syslog database, both must agree on its offsets across the stored history, and every value must exist in it. Stored and removed messages are then renamed and copied in verified batches in that zone, and a retry must reuse the same zone * issue#472: Purge Syslog Devices (Utilities) crashed with "Call to undefined function syslog_db_execute()" because the hook never connected to the Syslog database; it now connects first, reports connection failures, qualifies the Syslog database, and uses NOT EXISTS so a NULL host_id no longer prevents purging * issue#473: Syslog Status: choosing a card in Add card now appends it (Cacti's select2 dropdowns report changes through jQuery, which the card controller's native listener missed), and the Refresh interval is saved per user through a callback and refreshes the cards in place instead of reloading the page * issue#473: Bound every log query at the current time so empty future partitions are always pruned, whatever the filter's own date range diff --git a/README.md b/README.md index 8b1a81ba..a2f1b46c 100644 --- a/README.md +++ b/README.md @@ -240,25 +240,45 @@ columns, including `replication_source_event_id`, are copied unchanged. Existing `TIMESTAMP` rows are copied in UTC, preserving distinct events across daylight-saving clock changes. -Legacy `DATETIME` `logtime` columns are **not supported**. A `DATETIME` holds -wall-clock time in an unknown zone; values in a daylight-saving gap do not -exist as `TIMESTAMP` and values in the repeated hour are ambiguous, so no -automatic conversion is safe. The utility refuses to preserve such tables -before changing anything. Either rebuild them with `--discard-data`, or, after -verifying the zone the values were written in, convert them yourself with -Syslog processing stopped. First confirm that every value round-trips in -that zone, then convert and repartition in one statement: +Legacy `DATETIME` `logtime` columns, typical of installs created before +`TIMESTAMP` partitioning and often partitioned by `TO_DAYS(logtime)`, hold +wall-clock time with no zone. They cannot be widened in place: `TO_DAYS()` on a +`TIMESTAMP` column is rejected with "Constant, random or timezone-dependent +expressions in (sub)partitioning function are not allowed". The utility refuses +to preserve them unless you name the zone the values were written in, usually +the zone of the host that wrote them (rsyslog or the Cacti server): -```sql -SET SESSION time_zone = 'America/New_York'; -- the zone the values were written in -SELECT COUNT(*) FROM syslog - WHERE NOT (FROM_UNIXTIME(UNIX_TIMESTAMP(logtime)) <=> logtime); -- must be 0 -ALTER TABLE syslog MODIFY COLUMN logtime timestamp NOT NULL DEFAULT '0000-00-00 00:00:00' - PARTITION BY RANGE (UNIX_TIMESTAMP(logtime)) (/* daily partitions */ - PARTITION dMaxValue VALUES LESS THAN MAXVALUE); +```console +php plugins/syslog/cli/upgrade_partitioning.php --convert-ms --source-time-zone=America/New_York ``` -Then run this utility to apply the configured layout. +Before anything changes, `--source-time-zone` must pass these checks: + +* The zone is an IANA name known to PHP, and therefore to Cacti. +* The Syslog database server has it loaded in its time zone tables, which the + Cacti installer already requires (`mariadb-tzinfo-to-sql /usr/share/zoneinfo | + mariadb mysql`). +* PHP and the database agree on its UTC offset at every daylight-saving + transition spanning the stored history and now, and at both ends of every + day that has stored messages, so a transition missing from either side's + zone data (stale tzdata) is caught. +* Every `DATETIME` value exists in that zone. A value in a daylight-saving gap, + such as 02:30 on a spring-forward day, or outside the `TIMESTAMP` range means + the zone is probably wrong, and the command stops. + +The utility also prints Cacti's own zone, warns when it differs, and reports how +many rows fall in a repeated fall-back hour. Those values are ambiguous in the +source; MariaDB uses the first (daylight-time) occurrence. + +`syslog` and `syslog_removed` are then renamed and copied like any other +rebuild: the replacement is created with `TIMESTAMP` and `UNIX_TIMESTAMP` +partitions, and rows are copied with the session in the source zone. Each batch +checksum compares the same wall-clock text on both sides. The incoming queue +is converted in place, in the source zone, to keep its sequence identities, and +is verified unchanged afterwards. The zone is recorded until the rebuild +finishes, and a retry with a different `--source-time-zone` is refused. +Without the option, use `--discard-data` to replace legacy stored and removed +history. Every existing message is copied, including older-than-retention messages; those older rows fit in the earliest RANGE partition. Normal retention pruning @@ -303,7 +323,7 @@ changes, not routine rotation. only after every batch and the total row counts match. * A failure leaves originals in the live or `_rebuild` tables. Keep writers stopped, fix the reported error, then retry with `--preserve-data` and the - same `--convert-ms` option, if used. A retry deletes replacement rows copied + same `--convert-ms` and `--source-time-zone` options, if used. A retry deletes replacement rows copied after the last recorded batch and resumes from there. Do not change settings or manually delete `_rebuild` or `syslog_rebuild_progress` tables during recovery. @@ -900,7 +920,8 @@ described under Installation. Partitioning Level, engine, retention and future horizon, preserving messages by default. `--partition=hourly|daily` saves and applies a level. `--convert-ms` also converts the message pipeline to millisecond timestamps on -MariaDB. It reports database size and warns about additional disk space and +MariaDB. `--source-time-zone=` preserves legacy `DATETIME` history by +naming the zone it was written in. It reports database size and warns about additional disk space and potential several-hour duration before confirmation. Both utilities provide Cacti-style `--help` and `--version` output. diff --git a/cli/upgrade_partitioning.php b/cli/upgrade_partitioning.php index 9d9d3bce..ee295f4c 100644 --- a/cli/upgrade_partitioning.php +++ b/cli/upgrade_partitioning.php @@ -34,8 +34,20 @@ $preserve_data = true; $convert_ms = false; $partition = null; +$source_zone = null; foreach ($parms as $arg) { + if (strpos($arg, '--source-time-zone=') === 0) { + $source_zone = substr($arg, 19); + + if ($source_zone === '') { + fwrite(STDERR, 'ERROR: --source-time-zone needs a time zone name such as America/New_York.' . PHP_EOL); + exit(1); + } + + continue; + } + if (strpos($arg, '--partition=') === 0) { $levels = ['hourly' => 'hour', 'hour' => 'hour', 'daily' => 'day', 'day' => 'day']; $partition = $levels[strtolower(substr($arg, 12))] ?? null; @@ -102,16 +114,19 @@ exit(1); } -exit(syslog_upart_run($confirm, $convert_ms, $preserve_data, $partition) ? 0 : 1); +exit(syslog_upart_run($confirm, $convert_ms, $preserve_data, $partition, $source_zone) ? 0 : 1); /** * Rebuild partitions using the configured level, engine, retention and horizon. * DDL is not transactional; failed copies retain originals for a safe retry. * - * @param string|null $partition 'hour' or 'day' from --partition, saved to - * Settings once confirmed; null uses Settings. + * @param string|null $partition 'hour' or 'day' from --partition, saved to + * Settings once confirmed; null uses Settings. + * @param string|null $source_zone The zone legacy DATETIME logtime values were + * written in (--source-time-zone); null refuses them. */ -function syslog_upart_run(bool $confirm, bool $convert_ms, bool $preserve_data = true, ?string $partition = null): bool { +function syslog_upart_run(bool $confirm, bool $convert_ms, bool $preserve_data = true, ?string $partition = null, + ?string $source_zone = null): bool { global $syslogdb_default; $mariadb = syslog_db_is_mariadb(); @@ -179,7 +194,7 @@ function syslog_upart_run(bool $confirm, bool $convert_ms, bool $preserve_data = return false; } - return syslog_upart_run_locked($confirm, $convert_ms, $preserve_data, $engines[strtolower($engine)], (int) $days, $ahead, $granularity, $limit, $partition !== null); + return syslog_upart_run_locked($confirm, $convert_ms, $preserve_data, $engines[strtolower($engine)], (int) $days, $ahead, $granularity, $limit, $partition !== null, $source_zone); } finally { syslog_db_fetch_cell_prepared('SELECT RELEASE_LOCK(?)', [$lock]); } @@ -206,11 +221,185 @@ function syslog_upart_live_workers(): int { return $live; } +/** + * Validate --source-time-zone before anything changes. The zone must be an + * IANA name known to PHP (and so to Cacti) and loaded in the Syslog database + * server's time zone tables. PHP and the database must agree on its UTC offset + * at every transition spanning the stored history, so neither side's zone data + * is stale. Every legacy DATETIME value must exist in the zone (not fall in a + * daylight-saving gap) and fit the TIMESTAMP range. + * + * @param string $zone The zone the legacy DATETIME values were written in. + * @param string[] $tables Tables whose logtime column is DATETIME. + * + * @return string An error message, or '' when the zone may be used. + */ +function syslog_upart_source_zone_error(string $zone, array $tables): string { + global $syslogdb_default, $config; + + if (!in_array($zone, DateTimeZone::listIdentifiers(DateTimeZone::ALL_WITH_BC), true)) { + return "--source-time-zone=$zone is not a time zone known to PHP and Cacti; use an IANA name such as America/New_York."; + } + + $known = syslog_db_fetch_cell_prepared("SELECT CONVERT_TZ('2000-01-01 00:00:00', '+00:00', ?) IS NOT NULL", [$zone]); + + if ((string) $known !== '1') { + return "The Syslog database server does not know time zone $zone. Load its time zone tables, as the Cacti installer requires (for example: mariadb-tzinfo-to-sql /usr/share/zoneinfo | mariadb mysql), then retry."; + } + + $tz = new DateTimeZone($zone); + $now = time(); + $low = $now; + $high = $now; + $days = []; + + foreach ($tables as $table) { + $rows = syslog_db_fetch_assoc("SELECT DISTINCT DATE(logtime) AS day + FROM `$syslogdb_default`.`$table` WHERE logtime > '1970-01-01 00:00:00'"); + + if (!is_array($rows)) { + return "Cannot read the logtime range of $table."; + } + + foreach ($rows as $row) { + $days[(string) $row['day']] = true; + } + } + + // PHP's transitions alone would miss a transition only the database has + // (stale tzdata), so every stored day is probed at both ends, and days on + // which PHP's offset changes every 15 minutes. + $probes = [$now]; + + foreach (array_keys($days) as $day) { + $start = (new DateTime("$day 00:00:00", $tz))->getTimestamp(); + $end = (new DateTime("$day 23:59:59", $tz))->getTimestamp(); + $low = min($low, $start - 86400); + $high = max($high, $end + 86400); + $probes[] = $start; + $probes[] = $end; + + if ($tz->getOffset(new DateTime('@' . $start)) !== $tz->getOffset(new DateTime('@' . $end))) { + for ($probe = $start; $probe < $end; $probe += 900) { + $probes[] = $probe; + } + } + } + + // CONVERT_TZ leaves values outside the TIMESTAMP range unconverted. + $transitions = $tz->getTransitions(max(1, $low), min(2147483647, $high)) ?: []; + + foreach ($transitions as $transition) { + $probes[] = $transition['ts'] - 1; + $probes[] = $transition['ts']; + } + + $probes = array_values(array_filter(array_unique($probes), static fn($probe) => $probe >= 1 && $probe <= 2147483647)); + + foreach (array_chunk($probes, 500) as $chunk) { + $expected = []; + + foreach ($chunk as $probe) { + $expected[gmdate('Y-m-d H:i:s', $probe)] = (new DateTime('@' . $probe))->setTimezone($tz)->format('Y-m-d H:i:s'); + } + + $actual = syslog_db_fetch_assoc_prepared("SELECT CAST(probe.utc AS CHAR) AS utc, CAST(CONVERT_TZ(probe.utc, '+00:00', ?) AS CHAR) AS wall + FROM (" . implode(' UNION ALL ', array_fill(0, count($expected), 'SELECT CAST(? AS DATETIME) AS utc')) . ') AS probe', + array_merge([$zone], array_keys($expected))); + + if (!is_array($actual) || count($actual) !== count($expected)) { + return "Cannot compare $zone between PHP and the Syslog database."; + } + + foreach ($actual as $row) { + $utc = (string) $row['utc']; + + if (!isset($expected[$utc]) || (string) $row['wall'] !== $expected[$utc]) { + return "PHP and the Syslog database disagree on $zone at $utc UTC (PHP " . ($expected[$utc] ?? '?') . ', database ' . var_export($row['wall'], true) + . '). Update the database time zone tables or PHP time zone data so they match, then retry.'; + } + } + } + + $previous = syslog_db_fetch_cell('SELECT @@SESSION.time_zone'); + + if (!is_string($previous) || !syslog_db_execute_prepared('SET SESSION time_zone = ?', [$zone])) { + return "Cannot set the session time zone to $zone."; + } + + $ambiguous = 0; + + try { + foreach ($tables as $table) { + // Values in a daylight-saving gap or outside the TIMESTAMP range do not round-trip. + $invalid = syslog_db_fetch_row("SELECT COUNT(*) AS invalid_rows, MIN(logtime) AS sample + FROM `$syslogdb_default`.`$table` WHERE logtime <> '0000-00-00 00:00:00' + AND NOT (FROM_UNIXTIME(UNIX_TIMESTAMP(logtime)) <=> logtime)"); + + if (!is_array($invalid) || !isset($invalid['invalid_rows'])) { + return "Cannot check the logtime values of $table."; + } + + if ((int) $invalid['invalid_rows'] > 0) { + return "$table has {$invalid['invalid_rows']} logtime values, such as {$invalid['sample']}, that do not exist in $zone " + . "(a daylight-saving gap) or fall outside the TIMESTAMP range. $zone is probably not the zone they were written in; otherwise correct or delete those rows, then retry."; + } + + // When clocks fall back, the repeated wall-clock hour is ambiguous. + foreach ($transitions as $index => $transition) { + $offset = $index > 0 ? $transitions[$index - 1]['offset'] : $transition['offset']; + + if ($transition['offset'] >= $offset) { + continue; + } + + $start = (new DateTime('@' . $transition['ts']))->setTimezone($tz); + $end = (clone $start)->modify('+' . ($offset - $transition['offset']) . ' seconds'); + $count = syslog_db_fetch_cell_prepared("SELECT COUNT(*) FROM `$syslogdb_default`.`$table` + WHERE logtime >= ? AND logtime < ?", [$start->format('Y-m-d H:i:s'), $end->format('Y-m-d H:i:s')]); + + if (!is_numeric($count)) { + return "Cannot check the logtime values of $table."; + } + + $ambiguous += (int) $count; + } + } + } finally { + if (!syslog_db_execute_prepared('SET SESSION time_zone = ?', [$previous])) { + cacti_log('SYSLOG ERROR: Could not restore the session time zone after checking --source-time-zone', false, 'SYSLOG'); + } + } + + $cacti_zones = [date_default_timezone_get()]; + + if (db_column_exists('poller', 'timezone')) { + $cacti_zones[] = (string) db_fetch_cell_prepared('SELECT timezone FROM poller WHERE id = ?', [$config['poller_id'] ?? 1]); + } + + $cacti_zones = array_values(array_unique(array_filter($cacti_zones, static fn($name) => $name !== ''))); + + print "Source time zone $zone: known to PHP and the Syslog database, which agree on its offsets; every DATETIME value in " + . implode(', ', $tables) . ' exists in it.' . PHP_EOL; + + if ($ambiguous > 0) { + print "NOTE: $ambiguous rows fall in repeated daylight-saving hours; each converts to one of its two possible instants." . PHP_EOL; + } + + $mismatched = array_values(array_diff($cacti_zones, [$zone])); + + if ($mismatched) { + print 'WARNING: Cacti also uses time zone ' . implode(' and ', $mismatched) . ", not $zone. Confirm the legacy values were written in $zone." . PHP_EOL; + } + + return ''; +} + /** * Rebuild while holding the Syslog processing lock. */ function syslog_upart_run_locked(bool $confirm, bool $convert_ms, bool $preserve_data, string $engine, int $days, int $ahead, - string $granularity = 'day', int $limit = 8192, bool $save_level = false): bool { + string $granularity = 'day', int $limit = 8192, bool $save_level = false, ?string $source_zone = null): bool { global $syslogdb_default; if (syslog_db_table_exists('syslog_incoming_rebuild', false)) { @@ -237,10 +426,11 @@ function syslog_upart_run_locked(bool $confirm, bool $convert_ms, bool $preserve // The incoming queue and replication outbox keep their sequence identities // (archival and delivery deduplicate on them), so they are widened in place. + // A DATETIME(3) column is already millisecond but still zone-less. $widen = []; foreach (['syslog_incoming', 'syslog_replication_output'] as $table) { - if ($convert_ms && syslog_db_table_exists($table, false) && syslog_detect_logtime_precision($table) !== 'ms') { + if ($convert_ms && syslog_db_table_exists($table, false) && syslog_upart_logtime_type($table) !== 'timestamp(3)') { $widen[] = $table; } } @@ -251,8 +441,8 @@ function syslog_upart_run_locked(bool $confirm, bool $convert_ms, bool $preserve return true; } - // Legacy DATETIME history is wall-clock time in an unknown zone; it is - // not converted. Only an explicit discard may replace such tables. + // Legacy DATETIME history is wall-clock time; it is preserved only when + // --source-time-zone names the zone it was written in. $checked = $widen; foreach ($tables as $table) { @@ -274,13 +464,41 @@ function syslog_upart_run_locked(bool $confirm, bool $convert_ms, bool $preserve return false; } - if ($legacy) { - fwrite(STDERR, 'ERROR: Legacy DATETIME logtime columns are not supported: ' . implode(', ', array_column($legacy, 'TABLE_NAME')) - . '. Rebuild stored and removed history with --discard-data, or convert it manually as described in the README. No changes were made.' . PHP_EOL); + $legacy = array_column($legacy, 'TABLE_NAME'); + + if ($legacy && $source_zone === null) { + fwrite(STDERR, 'ERROR: Legacy DATETIME logtime columns found: ' . implode(', ', $legacy) + . '. Preserve them with --source-time-zone=, naming the zone their values were written in, or rebuild stored and removed history with --discard-data. No changes were made.' . PHP_EOL); + + return false; + } + + if (!$legacy && $source_zone !== null) { + print 'No legacy DATETIME logtime columns were found; --source-time-zone is ignored.' . PHP_EOL; + $source_zone = null; + } + + // A resumed copy must read the retained DATETIME original in the same zone. + $zone_setting = 'syslog_upart_source_time_zone'; + $resumed = array_values(array_filter($legacy, static fn($table) => substr($table, -8) === '_rebuild')); + $previous_zone = $resumed ? (string) read_config_option($zone_setting, true) : ''; + + if ($previous_zone !== '' && $previous_zone !== $source_zone) { + fwrite(STDERR, 'ERROR: ' . implode(', ', $resumed) . " was being copied from time zone $previous_zone. Retry with --source-time-zone=$previous_zone. No changes were made." . PHP_EOL); return false; } + if ($legacy) { + $zone_error = syslog_upart_source_zone_error($source_zone, $legacy); + + if ($zone_error !== '') { + fwrite(STDERR, "ERROR: $zone_error No changes were made." . PHP_EOL); + + return false; + } + } + $bytes = syslog_db_fetch_cell_prepared('SELECT SUM(DATA_LENGTH + INDEX_LENGTH) FROM information_schema.TABLES WHERE TABLE_SCHEMA = ?', [$syslogdb_default]); @@ -304,6 +522,11 @@ function syslog_upart_run_locked(bool $confirm, bool $convert_ms, bool $preserve : 'DESTRUCTIVE: stored messages in the listed rebuild tables will be DELETED.' . PHP_EOL; print $convert_ms ? 'Converting the complete message pipeline to TIMESTAMP(3).' . PHP_EOL : 'Existing timestamp precision is preserved.' . PHP_EOL; + + if ($source_zone !== null) { + print 'Legacy DATETIME values in ' . implode(', ', $legacy) . " are read as $source_zone wall-clock time." . PHP_EOL; + } + print 'Take a database backup. Syslog processing is disabled and its lock' . PHP_EOL; print 'is held, but this utility does NOT pause remote collectors, replication or' . PHP_EOL; print 'external database writers; stop them and buffer new logs upstream.' . PHP_EOL; @@ -326,6 +549,10 @@ function syslog_upart_run_locked(bool $confirm, bool $convert_ms, bool $preserve set_config_option('syslog_partition_granularity', $granularity); } + if ($source_zone !== null) { + set_config_option($zone_setting, $source_zone); + } + $zone = syslog_db_fetch_cell('SELECT @@SESSION.time_zone'); $mode = syslog_db_fetch_cell('SELECT @@SESSION.sql_mode'); $lock_wait = syslog_db_fetch_cell('SELECT @@SESSION.lock_wait_timeout'); @@ -349,15 +576,21 @@ function syslog_upart_run_locked(bool $confirm, bool $convert_ms, bool $preserve } foreach ($tables as $table) { - if (!syslog_upart_rebuild($table, $convert_ms, $preserve_data, $engine, $days, $ahead, $granularity, $limit)) { + if (!syslog_upart_rebuild($table, $convert_ms, $preserve_data, $engine, $days, $ahead, $granularity, $limit, 50000, $source_zone)) { return false; } } $column = syslog_logtime_sql_type('ms'); + if ($source_zone !== null && !syslog_db_execute_prepared('SET SESSION time_zone = ?', ['+00:00'])) { + fwrite(STDERR, 'ERROR: Cannot reset the copy session time zone; the precision setting was not changed.' . PHP_EOL); + + return false; + } + foreach ($widen as $table) { - if (!syslog_db_execute("ALTER TABLE `$syslogdb_default`.`$table` MODIFY COLUMN logtime $column")) { + if (!syslog_upart_widen($table, $column, in_array($table, $legacy, true) ? $source_zone : null)) { fwrite(STDERR, "ERROR: Cannot widen $table; the precision setting was not changed." . PHP_EOL); return false; @@ -366,7 +599,7 @@ function syslog_upart_run_locked(bool $confirm, bool $convert_ms, bool $preserve if ($convert_ms) { foreach (['syslog', 'syslog_removed', 'syslog_incoming', 'syslog_replication_output'] as $table) { - if (syslog_db_table_exists($table, false) && syslog_detect_logtime_precision($table) !== 'ms') { + if (syslog_db_table_exists($table, false) && syslog_upart_logtime_type($table) !== 'timestamp(3)') { fwrite(STDERR, "ERROR: $table did not convert; the precision setting was not changed." . PHP_EOL); return false; @@ -385,12 +618,69 @@ function syslog_upart_run_locked(bool $confirm, bool $convert_ms, bool $preserve } } - cacti_log('SYSLOG: Completed cli/upgrade_partitioning.php' . ($convert_ms ? ' with --convert-ms' : ''), false, 'SYSLOG'); + if ($source_zone !== null) { + set_config_option($zone_setting, ''); + } + + cacti_log('SYSLOG: Completed cli/upgrade_partitioning.php' . ($convert_ms ? ' with --convert-ms' : '') + . ($source_zone !== null ? " from DATETIME in $source_zone" : ''), false, 'SYSLOG'); print 'Rebuild complete. Re-enable Syslog processing.' . ($convert_ms ? ' Syslog now stores millisecond timestamps; configure forwarders accordingly.' : '') . PHP_EOL; return true; } +/** + * Return the lowercase logtime column type of a table, such as timestamp(3). + */ +function syslog_upart_logtime_type(string $table): string { + global $syslogdb_default; + + return strtolower((string) syslog_db_fetch_cell_prepared('SELECT COLUMN_TYPE FROM information_schema.COLUMNS + WHERE TABLE_SCHEMA = ? AND TABLE_NAME = ? AND COLUMN_NAME = \'logtime\'', [$syslogdb_default, $table])); +} + +/** + * Widen an incoming queue or outbox logtime column in place, so the sequence + * identities that archival and replication rely on are kept. A legacy + * DATETIME column is converted in its source zone, and its wall-clock values + * are verified unchanged when read back in that zone. + * + * @param string $table The table to widen. + * @param string $column The new logtime column definition. + * @param string|null $source_zone The zone of a legacy DATETIME column, or null. + */ +function syslog_upart_widen(string $table, string $column, ?string $source_zone): bool { + global $syslogdb_default; + + $target = "`$syslogdb_default`.`$table`"; + + if ($source_zone === null) { + return syslog_db_execute("ALTER TABLE $target MODIFY COLUMN logtime $column"); + } + + $digest = "SELECT COUNT(*) AS row_count, SUM(CRC32(DATE_FORMAT(logtime, '%Y-%m-%d %H:%i:%s.%f'))) AS digest FROM $target"; + + if (!syslog_db_execute_prepared('SET SESSION time_zone = ?', [$source_zone])) { + return false; + } + + try { + $before = syslog_db_fetch_row($digest); + + if (!is_array($before) || !isset($before['row_count']) + || !syslog_db_execute("ALTER TABLE $target MODIFY COLUMN logtime $column")) { + return false; + } + + $after = syslog_db_fetch_row($digest); + + return is_array($after) && (string) ($after['row_count'] ?? '') === (string) $before['row_count'] + && (string) ($after['digest'] ?? '') === (string) $before['digest']; + } finally { + syslog_db_execute("SET SESSION time_zone = '+00:00'"); + } +} + /** * Rebuild one message table without dropping its original until data agrees. * @@ -401,9 +691,14 @@ function syslog_upart_run_locked(bool $confirm, bool $convert_ms, bool $preserve * retry discards replacement rows above the last checkpoint and resumes after it. * At 10M rows on a 2 GiB buffer pool, batches of 10,000 to 65,536 rows ran * within measurement noise of each other; 50,000 was fastest. + * + * A legacy DATETIME original is copied with the session in $source_zone, so its + * wall-clock values become the matching UTC instants, and each batch checksum + * compares the same wall-clock text on both sides. TIMESTAMP originals are + * copied in UTC, which keeps daylight-saving repeated hours distinct. */ function syslog_upart_rebuild(string $table, bool $convert_ms, bool $preserve_data, string $engine, int $days, int $ahead, - string $granularity = 'day', int $limit = 8192, int $batch = 50000): bool { + string $granularity = 'day', int $limit = 8192, int $batch = 50000, ?string $source_zone = null): bool { global $syslogdb_default; $batch = max(1, $batch); @@ -417,7 +712,7 @@ function syslog_upart_rebuild(string $table, bool $convert_ms, bool $preserve_da $progress = "`$syslogdb_default`.`syslog_rebuild_progress`"; $fail = static function (string $reason) use ($table, $backup): bool { - fwrite(STDERR, "ERROR: $table: $reason. Originals are retained in $table or $backup; keep writers stopped and retry with --preserve-data and the same --convert-ms option, if used. The precision setting was not changed." . PHP_EOL); + fwrite(STDERR, "ERROR: $table: $reason. Originals are retained in $table or $backup; keep writers stopped and retry with --preserve-data and the same --convert-ms and --source-time-zone options, if used. The precision setting was not changed." . PHP_EOL); return false; }; @@ -448,8 +743,8 @@ function syslog_upart_rebuild(string $table, bool $convert_ms, bool $preserve_da $logtime = array_values(array_filter($columns, static fn($column) => $column['COLUMN_NAME'] === 'logtime')); $type = $logtime ? $logtime[0]['DATA_TYPE'] : ''; - if ($type === 'datetime' && $preserve_data) { - return $fail('Legacy DATETIME logtime is not supported; rebuild with --discard-data or convert it manually'); + if ($type === 'datetime' && $preserve_data && $source_zone === null) { + return $fail('Legacy DATETIME logtime needs --source-time-zone to be preserved; otherwise rebuild with --discard-data'); } if (!in_array($type, ['timestamp', 'datetime'], true)) { @@ -457,7 +752,9 @@ function syslog_upart_rebuild(string $table, bool $convert_ms, bool $preserve_da } // TIMESTAMP is stored in UTC; a UTC session copies it without conversion. - if (!syslog_db_execute("SET SESSION time_zone = '+00:00'")) { + $copy_zone = $type === 'datetime' && $source_zone !== null ? $source_zone : '+00:00'; + + if (!syslog_db_execute_prepared('SET SESSION time_zone = ?', [$copy_zone])) { return $fail('Cannot configure the timestamp copy time zone'); } @@ -802,7 +1099,8 @@ function syslog_upart_display_version(): void { function syslog_upart_display_help(): void { syslog_upart_display_version(); - print PHP_EOL . 'usage: upgrade_partitioning.php [--partition=hourly|daily] [--convert-ms] [--preserve-data | --discard-data] [--confirm]' . PHP_EOL . PHP_EOL; + print PHP_EOL . 'usage: upgrade_partitioning.php [--partition=hourly|daily] [--convert-ms] [--preserve-data | --discard-data]' . PHP_EOL; + print ' [--source-time-zone=] [--confirm]' . PHP_EOL . PHP_EOL; print 'Rebuilds syslog and syslog_removed with UTC hourly or daily partitions matching' . PHP_EOL; print 'the Partitioning Level, storage engine, message retention and future-partition' . PHP_EOL; print 'horizon. A Partitioning Level changed in Settings takes effect only when this' . PHP_EOL; @@ -811,12 +1109,15 @@ function syslog_upart_display_help(): void { print 'With preservation, rows are copied in verified batches and receive new sequence' . PHP_EOL; print 'numbers above the original high-water mark; originals are dropped only after' . PHP_EOL; print 'every batch and the row counts are verified.' . PHP_EOL . PHP_EOL; - print 'Legacy DATETIME logtime columns are not converted; use --discard-data for them.' . PHP_EOL . PHP_EOL; + print 'Legacy DATETIME logtime columns are preserved only with --source-time-zone,' . PHP_EOL; + print 'naming the zone their values were written in; otherwise use --discard-data.' . PHP_EOL . PHP_EOL; print 'Options:' . PHP_EOL; print ' --partition - Save and apply this Partitioning Level instead of the one in Settings.' . PHP_EOL; print ' --convert-ms - Also convert pipeline timestamps to TIMESTAMP(3); MariaDB only.' . PHP_EOL; print ' --preserve-data - Preserve existing messages (the default), copying 50,000 rows at a time.' . PHP_EOL; print ' --discard-data - Delete messages in rebuilt tables; never discard the outbox.' . PHP_EOL; + print ' --source-time-zone - IANA zone (for example America/New_York) of legacy DATETIME values.' . PHP_EOL; + print ' It must be known to PHP and the database, which must agree on its offsets.' . PHP_EOL; print ' --confirm - Skip confirmation, not the size and maintenance warnings.' . PHP_EOL; print ' --version - Display version information.' . PHP_EOL; print ' --help - Display this help.' . PHP_EOL . PHP_EOL; @@ -826,5 +1127,5 @@ function syslog_upart_display_help(): void { print 'much ADDITIONAL free disk space on the database server, plus temporary' . PHP_EOL; print 'file and binary-log headroom. Large transfers may take several hours.' . PHP_EOL . PHP_EOL; print 'On failure keep writers stopped and retry with --preserve-data and the' . PHP_EOL; - print 'same --partition and --convert-ms options, if used; originals remain in *_rebuild tables.' . PHP_EOL . PHP_EOL; + print 'same --partition, --convert-ms and --source-time-zone options, if used; originals remain in *_rebuild tables.' . PHP_EOL . PHP_EOL; } diff --git a/tests/Unit/PartitionRebuildTest.php b/tests/Unit/PartitionRebuildTest.php index 699211bb..79d7de53 100644 --- a/tests/Unit/PartitionRebuildTest.php +++ b/tests/Unit/PartitionRebuildTest.php @@ -8,13 +8,16 @@ function partition_rebuild_runner(): string { preg_match('/^function syslog_upart_run\(.*?^\}/ms', $source, $run); preg_match('/^function syslog_upart_run_locked\(.*?^\}/ms', $source, $locked); preg_match('/^function syslog_upart_live_workers\(.*?^\}/ms', $source, $workers); + preg_match('/^function syslog_upart_widen\(.*?^\}/ms', $source, $widen); + preg_match('/^function syslog_upart_logtime_type\(.*?^\}/ms', $source, $type); $path = tempnam(sys_get_temp_dir(), 'syslog-rebuild-unit-'); file_put_contents($path, ' [], 'legacy' => [], 'retention' => '30', - 'granularity' => '' + 'granularity' => '', + 'stored_zone' => '', + 'column_type' => null ]; $GLOBALS['config']['poller_id'] = 1; test_override('read_config_option', static fn($key) => [ 'syslog_install_engine' => 'InnoDB', 'syslog_retention' => $state->retention, 'syslog_partition_ahead_days' => '3', 'syslog_enabled' => $state->enabled, - 'syslog_partition_granularity' => $state->granularity + 'syslog_partition_granularity' => $state->granularity, + 'syslog_upart_source_time_zone' => $state->stored_zone ][$key] ?? ''); test_override('syslog_db_fetch_assoc_prepared', static function ($sql) use ($state) { $state->assoc[] = $sql; @@ -88,7 +94,7 @@ function partition_rebuild_setup(): stdClass { } if (str_contains($sql, 'COLUMN_TYPE')) { - return $state->precision === 'ms' ? 'timestamp(3)' : 'timestamp'; + return $state->column_type ?? ($state->precision === 'ms' ? 'timestamp(3)' : 'timestamp'); } return $state->bytes; @@ -117,7 +123,7 @@ function partition_rebuild_setup(): stdClass { ->toContain('does NOT pause')->toContain('50,000-row copies') ->toContain('receive new sequence numbers') ->not->toContain('Type "yes"') - ->and($state->rebuilds)->toBe([['syslog', false, true, 'InnoDB', 30, 3, 'day', 8192]]) + ->and($state->rebuilds)->toBe([['syslog', false, true, 'InnoDB', 30, 3, 'day', 8192, 50000, null]]) ->and($state->queries[0])->toBe(['SELECT GET_LOCK(?, 0)', [syslog_processing_lock_name()]]); expect($state->settings)->toBe([]); expect(array_column($state->queries, 0))->toContain('SELECT RELEASE_LOCK(?)'); @@ -141,7 +147,7 @@ function partition_rebuild_setup(): stdClass { expect($ok)->toBeTrue() ->and($output)->toContain('Rebuilding hourly partitions') - ->and($state->rebuilds)->toBe([['syslog', false, true, 'InnoDB', 30, 2, 'hour', 8192]]); + ->and($state->rebuilds)->toBe([['syslog', false, true, 'InnoDB', 30, 2, 'hour', 8192, 50000, null]]); }); it('refuses hourly partitioning beyond the server partition limit or with indefinite retention', function () { @@ -221,6 +227,114 @@ function partition_rebuild_setup(): stdClass { ->and(end($state->queries))->toBe(['SELECT RELEASE_LOCK(?)', [syslog_processing_lock_name()]]); }); +it('refuses a source time zone that fails validation before any change', function () { + $state = partition_rebuild_setup(); + $state->legacy = [['TABLE_NAME' => 'syslog']]; + test_override('syslog_upart_source_zone_error', static fn($zone, $tables) => "zone $zone rejected for " . implode(',', $tables)); + $run = partition_rebuild_runner(); + + expect($run(true, false, true, null, 'Mars/Olympus'))->toBeFalse() + ->and($state->rebuilds)->toBe([]) + ->and($state->statements)->toBe([]) + ->and($state->settings)->toBe([]) + ->and(end($state->queries))->toBe(['SELECT RELEASE_LOCK(?)', [syslog_processing_lock_name()]]); +}); + +it('copies legacy DATETIME history in the validated source zone and clears the retry zone afterwards', function () { + $state = partition_rebuild_setup(); + $state->legacy = [['TABLE_NAME' => 'syslog']]; + $checked = []; + test_override('syslog_upart_source_zone_error', static function ($zone, $tables) use (&$checked) { + $checked[] = [$zone, $tables]; + + return ''; + }); + test_override('set_config_option', static function ($key, $value) use ($state) { + $state->settings[] = [$key, $value]; + }); + $run = partition_rebuild_runner(); + ob_start(); + $ok = $run(true, false, true, null, 'America/New_York'); + $output = ob_get_clean(); + + expect($ok)->toBeTrue() + ->and($checked)->toBe([['America/New_York', ['syslog']]]) + ->and($output)->toContain('read as America/New_York wall-clock time') + ->and($state->rebuilds)->toBe([['syslog', false, true, 'InnoDB', 30, 3, 'day', 8192, 50000, 'America/New_York']]) + ->and($state->settings)->toBe([['syslog_upart_source_time_zone', 'America/New_York'], ['syslog_upart_source_time_zone', '']]); +}); + +it('refuses to resume an interrupted DATETIME copy in a different source zone', function () { + $state = partition_rebuild_setup(); + $state->tables[] = 'syslog_rebuild'; + $state->legacy = [['TABLE_NAME' => 'syslog_rebuild']]; + $state->stored_zone = 'Europe/Berlin'; + $run = partition_rebuild_runner(); + + expect($run(true, false, true, null, 'America/New_York'))->toBeFalse() + ->and($state->rebuilds)->toBe([]) + ->and($state->settings)->toBe([]); +}); + +it('converts a legacy DATETIME incoming queue in place in the source zone', function () { + $state = partition_rebuild_setup(); + $state->tables = ['syslog_incoming']; + $state->legacy = [['TABLE_NAME' => 'syslog_incoming']]; + test_override('syslog_db_fetch_row', static fn($sql) => str_contains($sql, 'CRC32') + ? ['row_count' => '2', 'digest' => '99'] : ['Value' => $state->version]); + test_override('syslog_db_execute', static function ($sql) use ($state) { + $state->statements[] = $sql; + $state->precision = str_starts_with($sql, 'ALTER TABLE') ? 'ms' : $state->precision; + + return true; + }); + $run = partition_rebuild_runner(); + ob_start(); + $ok = $run(true, true, true, null, 'America/New_York'); + ob_end_clean(); + + expect($ok)->toBeTrue() + ->and($state->rebuilds)->toBe([]) + ->and($state->executions)->toContain(['SET SESSION time_zone = ?', ['America/New_York']]) + ->and($state->statements)->toContain("ALTER TABLE `fixture`.`syslog_incoming` MODIFY COLUMN logtime timestamp(3) NOT NULL DEFAULT '0000-00-00 00:00:00.000'"); +}); + +it('converts a millisecond DATETIME(3) incoming queue to TIMESTAMP(3) instead of skipping it', function () { + $state = partition_rebuild_setup(); + $state->tables = ['syslog_incoming']; + $state->legacy = [['TABLE_NAME' => 'syslog_incoming']]; + $state->column_type = 'datetime(3)'; + test_override('syslog_db_fetch_row', static fn($sql) => str_contains($sql, 'CRC32') + ? ['row_count' => '1', 'digest' => '7'] : ['Value' => $state->version]); + test_override('syslog_db_execute', static function ($sql) use ($state) { + $state->statements[] = $sql; + $state->column_type = str_starts_with($sql, 'ALTER TABLE') ? 'timestamp(3)' : $state->column_type; + + return true; + }); + $run = partition_rebuild_runner(); + ob_start(); + $ok = $run(true, true, true, null, 'America/New_York'); + ob_end_clean(); + + expect($ok)->toBeTrue() + ->and($state->statements)->toContain("ALTER TABLE `fixture`.`syslog_incoming` MODIFY COLUMN logtime timestamp(3) NOT NULL DEFAULT '0000-00-00 00:00:00.000'"); + + // Without the conversion the final type check refuses to record success. + $state = partition_rebuild_setup(); + $state->tables = ['syslog_incoming']; + $state->legacy = [['TABLE_NAME' => 'syslog_incoming']]; + $state->column_type = 'datetime(3)'; + test_override('syslog_db_fetch_row', static fn($sql) => str_contains($sql, 'CRC32') + ? ['row_count' => '1', 'digest' => '7'] : ['Value' => $state->version]); + ob_start(); + $ok = $run(true, true, true, null, 'America/New_York'); + ob_end_clean(); + + expect($ok)->toBeFalse() + ->and($state->settings)->toBe(['syslog_upart_source_time_zone' => 'America/New_York']); +}); + it('discards without a column-type query when no table needs checking', function () { $state = partition_rebuild_setup(); $run = partition_rebuild_runner(); @@ -230,7 +344,7 @@ function partition_rebuild_setup(): stdClass { expect($ok)->toBeTrue() ->and(array_filter($state->assoc, static fn($sql) => str_contains($sql, 'information_schema.COLUMNS')))->toBe([]) - ->and($state->rebuilds)->toBe([['syslog', false, false, 'InnoDB', 30, 3, 'day', 8192]]); + ->and($state->rebuilds)->toBe([['syslog', false, false, 'InnoDB', 30, 3, 'day', 8192, 50000, null]]); }); it('widens the incoming queue in place so its sequence identities are kept', function () { @@ -335,7 +449,8 @@ function partition_rebuild_setup(): stdClass { 'Syslog Partition Upgrade Utility, Version 5.0, Copyright (C)', PHP_EOL . PHP_EOL . 'usage: upgrade_partitioning.php', 'row counts are verified.' . PHP_EOL . PHP_EOL . 'Legacy DATETIME', - '--discard-data for them.' . PHP_EOL . PHP_EOL . 'Options:', + 'otherwise use --discard-data.' . PHP_EOL . PHP_EOL . 'Options:', + '--source-time-zone - IANA zone', 'several hours.' . PHP_EOL . PHP_EOL . 'On failure', 'writers first.' . PHP_EOL . PHP_EOL . 'The current database size' ]], diff --git a/tests/regression/partition_rebuild_database_test.php b/tests/regression/partition_rebuild_database_test.php index 3c984dea..bbc9d4e1 100644 --- a/tests/regression/partition_rebuild_database_test.php +++ b/tests/regression/partition_rebuild_database_test.php @@ -77,6 +77,7 @@ function syslog_db_fetch_row_prepared($sql, $params = [], ...$args) { return $rows[0] ?? []; } +function syslog_db_fetch_assoc($sql, ...$args) { return syslog_db_fetch_assoc_prepared($sql); } function syslog_db_fetch_row($sql) { return syslog_db_fetch_row_prepared($sql); } function syslog_db_fetch_cell_prepared($sql, $params = [], ...$args) { $query = $GLOBALS['pdo']->prepare($sql); @@ -90,6 +91,9 @@ function syslog_db_table_exists($table, ...$args) { WHERE TABLE_SCHEMA = ? AND TABLE_NAME = ?', [$GLOBALS['syslogdb_default'], $table]); } function syslog_partition_ahead_days() { return (int) read_config_option('syslog_partition_ahead_days'); } +// No Cacti poller table here; Cacti's zone comes from PHP alone. +function db_column_exists(...$args) { return false; } +function db_fetch_cell_prepared(...$args) { return ''; } // No Cacti process table here, so no transfer workers are registered. function db_fetch_assoc($sql, ...$args) { return []; } @@ -311,6 +315,108 @@ function rebuild_table(string $table, bool $convert = true, string $engine = 'In ob_end_clean(); rebuild_check((int) syslog_db_fetch_cell('SELECT COUNT(*) FROM syslog') === 0 && is_free_processing_lock(), 'The discard command empties the rebuilt tables and releases the lock'); + // Legacy installs: DATETIME logtime partitioned by TO_DAYS, which cannot be + // widened in place to TIMESTAMP. --source-time-zone copies it instead. + rebuild_check((int) syslog_db_fetch_cell('SELECT COUNT(*) FROM mysql.time_zone_name') > 0, 'The test server needs its time zone tables loaded'); + $pdo->exec('DROP TABLE syslog, syslog_removed, syslog_incoming, syslog_replication_output'); + $pdo->exec("CREATE TABLE syslog (seq BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, logtime DATETIME NOT NULL DEFAULT '0000-00-00 00:00:00', + message TEXT NOT NULL, PRIMARY KEY(seq,logtime), KEY logtime(logtime)) ENGINE=InnoDB + PARTITION BY RANGE (TO_DAYS(logtime)) (PARTITION d20250101 VALUES LESS THAN (TO_DAYS('2025-01-01')), PARTITION dMaxValue VALUES LESS THAN MAXVALUE)"); + $pdo->exec("INSERT INTO syslog (logtime,message) SELECT '2025-06-01 00:00:00' + INTERVAL seq MINUTE, CONCAT('Event ',seq) FROM seq_1_to_12000"); + $pdo->exec("INSERT INTO syslog (logtime,message) VALUES ('2025-03-09 01:59:59','before the gap'),('2025-03-09 03:00:00','after the gap'), + ('2025-11-02 01:30:00','repeated hour'),('2024-12-31 23:00:00','winter'),('0000-00-00 00:00:00','zero')"); + $pdo->exec('CREATE TABLE syslog_removed LIKE syslog'); + $pdo->exec("INSERT INTO syslog_removed (logtime,message) VALUES ('2025-07-04 09:15:00','removed')"); + $pdo->exec('CREATE TABLE syslog_incoming (seq BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY, logtime DATETIME(3) NOT NULL, status INT, message TEXT) ENGINE=InnoDB'); + $pdo->exec("INSERT INTO syslog_incoming VALUES (70,'2025-08-01 18:45:00',0,'queued')"); + $legacy_total = (int) syslog_db_fetch_cell('SELECT COUNT(*) FROM syslog'); + $new_york = new DateTimeZone('America/New_York'); + $epoch = static fn(string $wall): string => (string) (new DateTime($wall, $new_york))->getTimestamp() . '.000'; + $untouched = static fn(): bool => (int) syslog_db_fetch_cell("SELECT COUNT(*) FROM information_schema.COLUMNS WHERE TABLE_SCHEMA=DATABASE() + AND COLUMN_NAME='logtime' AND DATA_TYPE='datetime'") === 3 && !syslog_db_table_exists('syslog_rebuild'); + $options['syslog_install_logtime_precision'] = 'sec'; + $pdo->exec("SET SESSION time_zone = '+05:30'"); + + foreach ([[null, 'no zone'], ['Mars/Olympus', 'unknown to PHP'], ['Europe/Berlin', 'Berlin daylight-saving gap']] as [$zone, $reason]) { + if ($zone === 'Europe/Berlin') { + $pdo->exec("INSERT INTO syslog (logtime,message) VALUES ('2025-03-30 02:30:00','berlin gap')"); + } + + ob_start(); + rebuild_check(!syslog_upart_run(true, true, true, null, $zone), "Legacy DATETIME is refused: $reason"); + ob_end_clean(); + rebuild_check($untouched() && is_free_processing_lock(), "A $reason refusal leaves every DATETIME table untouched"); + } + + $pdo->exec("DELETE FROM syslog WHERE message='berlin gap'"); + $pdo->exec("INSERT INTO syslog (logtime,message) VALUES ('2025-03-09 02:30:00','new york gap')"); + ob_start(); + rebuild_check(!syslog_upart_run(true, true, true, null, 'America/New_York'), 'A value in the source zone daylight-saving gap is refused'); + ob_end_clean(); + rebuild_check($untouched(), 'A gap refusal leaves every DATETIME table untouched'); + $pdo->exec("DELETE FROM syslog WHERE message='new york gap'"); + + // Stale database tzdata: Mexico City abolished DST in 2022, but this server + // is given a 2025 summer DST period PHP does not have. No PHP transition + // falls in the stored range, so only per-day probes can catch it. + $mexico = (int) syslog_db_fetch_cell("SELECT Time_zone_id FROM mysql.time_zone_name WHERE Name='America/Mexico_City'"); + // OFFSET is reserved in MariaDB 10.6+, so the column must be quoted. + $types = array_column(syslog_db_fetch_assoc_prepared('SELECT Transition_type_id, `Offset` FROM mysql.time_zone_transition_type + WHERE Time_zone_id = ? AND `Offset` IN (-21600, -18000)', [$mexico]), 'Transition_type_id', 'Offset'); + rebuild_check(isset($types[-21600], $types[-18000]), 'Mexico City has standard and daylight transition types'); + $stale_transitions = [(new DateTime('2025-04-06 08:00:00 UTC'))->getTimestamp() => $types[-18000], (new DateTime('2025-10-26 07:00:00 UTC'))->getTimestamp() => $types[-21600]]; + + try { + foreach ($stale_transitions as $at => $type) { + $pdo->prepare('INSERT INTO mysql.time_zone_transition VALUES (?, ?, ?)')->execute([$mexico, $at, $type]); + } + + rebuild_check(syslog_db_fetch_cell("SELECT CONVERT_TZ('2025-07-01 17:00:00','+00:00','America/Mexico_City')") === '2025-07-01 12:00:00', 'The server applies the stale summer DST'); + ob_start(); + rebuild_check(!syslog_upart_run(true, true, true, null, 'America/Mexico_City'), 'A transition only the database has is refused'); + ob_end_clean(); + rebuild_check($untouched() && is_free_processing_lock(), 'A stale-tzdata refusal leaves every DATETIME table untouched'); + } finally { + foreach (array_keys($stale_transitions) as $at) { + $pdo->prepare('DELETE FROM mysql.time_zone_transition WHERE Time_zone_id = ? AND Transition_time = ?')->execute([$mexico, $at]); + } + } + + // Interrupt the copy, then prove a retry must reuse the original zone. + $copy_attempts = 0; + $fail_copy_at = 1; + ob_start(); + rebuild_check(!syslog_upart_run(true, true, true, null, 'America/New_York'), 'An injected copy failure stops the DATETIME rebuild'); + ob_end_clean(); + $fail_copy_at = 0; + rebuild_check(syslog_db_table_exists('syslog_rebuild') && $options['syslog_upart_source_time_zone'] === 'America/New_York', 'The DATETIME original and its source zone are retained'); + ob_start(); + rebuild_check(!syslog_upart_run(true, true, true, null, 'America/Chicago'), 'A retry in a different source zone is refused'); + ob_end_clean(); + rebuild_check(syslog_db_table_exists('syslog_rebuild'), 'A refused zone change keeps the original'); + ob_start(); + rebuild_check(syslog_upart_run(true, true, true, null, 'America/New_York'), 'A retry in the original source zone completes'); + $output = ob_get_clean(); + rebuild_check(str_contains($output, 'NOTE: 1 rows fall in repeated daylight-saving hours'), 'Repeated-hour rows are reported'); + rebuild_check(!syslog_db_table_exists('syslog_rebuild') && !syslog_db_table_exists('syslog_removed_rebuild'), 'Verified DATETIME originals are removed'); + rebuild_check($options['syslog_install_logtime_precision'] === 'ms' && $options['syslog_upart_source_time_zone'] === '', 'Precision is recorded and the retry zone cleared'); + rebuild_check((int) syslog_db_fetch_cell('SELECT COUNT(*) FROM syslog') === $legacy_total, 'Every legacy row is preserved'); + rebuild_check((int) syslog_db_fetch_cell("SELECT COUNT(*) FROM information_schema.COLUMNS WHERE TABLE_SCHEMA=DATABASE() + AND COLUMN_NAME='logtime' AND COLUMN_TYPE='timestamp(3)'") === 3, 'Every legacy table now uses TIMESTAMP(3)'); + rebuild_check(syslog_upart_layout_ok('syslog', 'InnoDB', 'day'), 'The TO_DAYS table is repartitioned by UNIX_TIMESTAMP'); + + foreach (['before the gap' => '2025-03-09 01:59:59', 'after the gap' => '2025-03-09 03:00:00', 'winter' => '2024-12-31 23:00:00', 'Event 1' => '2025-06-01 00:01:00'] as $message => $wall) { + rebuild_check(syslog_db_fetch_cell_prepared('SELECT UNIX_TIMESTAMP(logtime) FROM syslog WHERE message = ?', [$message]) === $epoch($wall), "'$message' keeps its New York instant"); + } + + rebuild_check(in_array(syslog_db_fetch_cell("SELECT UNIX_TIMESTAMP(logtime) FROM syslog WHERE message='repeated hour'"), + [$epoch('2025-11-02 01:30:00 EDT'), $epoch('2025-11-02 01:30:00 EST')], true), 'A repeated-hour value converts to one of its two instants'); + rebuild_check(syslog_db_fetch_cell("SELECT logtime FROM syslog WHERE message='zero'") === '0000-00-00 00:00:00.000', 'Zero timestamps survive'); + rebuild_check(syslog_db_fetch_cell('SELECT UNIX_TIMESTAMP(logtime) FROM syslog_removed') === $epoch('2025-07-04 09:15:00'), 'Removed history keeps its New York instant'); + rebuild_check((int) syslog_db_fetch_cell('SELECT seq FROM syslog_incoming') === 70 + && syslog_db_fetch_cell('SELECT UNIX_TIMESTAMP(logtime) FROM syslog_incoming') === $epoch('2025-08-01 18:45:00'), 'The incoming queue keeps its sequence and New York instant'); + rebuild_check(syslog_db_fetch_cell('SELECT @@SESSION.time_zone') === '+05:30', 'The original session zone is restored'); + print "$checks partition rebuild database checks passed.\n"; } finally { $pdo->exec('DROP DATABASE `syslog_rebuild_test`');