Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@
* issue: Mac Addresses are not displaying correctly
* issue: Mactrack does not shutdown properly due to incorrect use of unregister_process
* issue: Make the MacTrack tab script repoint only the MacTrack main tab rather than whichever main tab is selected
* issue: Partition 802.1x history on new installations and include it in partition maintenance and upgrades
* issue: Pin the CI database image and shared test dependencies, disable persisted checkout credentials, and limit workflow permissions and runtime
* issue: Reach PHPStan level 8 with zero errors across all first-party PHP and add native function return type declarations, using no baselines or inline suppressions
* issue: Relocate the per-theme override stylesheets from themes/<theme>/ to css/<theme>/ alongside the base css/mactrack.css
Expand Down
2 changes: 1 addition & 1 deletion INFO
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@

[info]
name = mactrack
version = 5.0
version = 5.1
longname = Device Tracking
author = The Cacti Group
email =
Expand Down
17 changes: 9 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,18 +53,19 @@ Add Mactrack Site - Console -> Mactrack -> Sites
Add/import devices - Console -> Mactrack -> Devices -> Add device or Management -> Choose host -> Import into Mactrack Database
Create new or enable prepared Device Types - Console -> Mactrack -> Device Types

## Port and IP history partitioning
## Port, IP and 802.1x history partitioning

MacTrack keeps the MAC address history in `mac_track_ports` and the IP address
history in `mac_track_ips`. Both tables are partitioned by scan date (MySQL 8.0+
history in `mac_track_ips`, and 802.1x events in `mac_track_dot1x`. These tables
are partitioned by scan date (MySQL 8.0+
or MariaDB with the partition plugin), so retention drops whole expired
partitions instead of deleting millions of rows, and a search for one scan reads
one partition. A server without partitioning support keeps unpartitioned tables,
which MacTrack fully supports.

### Setting it up

**New installs** create both tables weekly partitioned, starting Monday at
**New installs** create all three tables weekly partitioned, starting Monday at
00:00 UTC, and the poller's daily maintenance keeps the next two days of
partitions created ahead of collection. Nothing needs to be done.

Expand All @@ -76,8 +77,8 @@ the size, then run the rebuild once:
Scan* (maintained by the poller; the rebuild script cannot build these two),
or *Disabled* for unpartitioned tables. It sets the size of the partitions
created from now on and never rewrites the ones that exist.
2. Set the retention (**Data Retention** for MAC addresses and **Data Retention
IPs**). Partitions older than the retention are dropped by the daily
2. Set the retention (**Data Retention** for MAC addresses and 802.1x events,
and **Data Retention IPs** for IP addresses). Partitions older than the retention are dropped by the daily
maintenance, so with weekly partitions up to one extra week of history can
stay.
3. Rebuild the existing history to match, either with **MacTrack Utilities >
Expand All @@ -94,7 +95,7 @@ Run it as the user that runs the Cacti poller, off hours (see below).

### Checking and changing the scheme

`cli/upgrade_partitioning.php` checks both tables against the scheme and rebuilds
`cli/upgrade_partitioning.php` checks all three tables against the scheme and rebuilds
only the ones that have drifted:

- Without `--partition` the scheme is the **Port and IP Address Partition Size** setting. This
Expand All @@ -109,7 +110,7 @@ A table is reported as drifted, and rebuilt, when it is not partitioned, uses th
earlier daily layout, has partitions MacTrack does not recognize, mixes sizes,
keeps rows outside every partition, or is not the selected size. A table that
already follows the scheme is left alone, and the script says so. **View
Partition Status** in the Utilities shows the layout of both tables, the last
Partition Status** in the Utilities shows the layout of all three tables, the last
rebuild and the last maintenance.

### What a rebuild does
Expand All @@ -135,7 +136,7 @@ process registry for 24 hours, so two runs cannot overlap).
- Scheduled scans and the daily database maintenance are paused. If the
maintenance window passes meanwhile, the poller runs the maintenance as soon as
the rebuild is over.
- MAC address and IP address searching shows a maintenance-mode message, and its
- MAC address, IP address and 802.1x searching shows a maintenance-mode message, and its
results are incomplete until the rebuild has finished.
- Changes to the history are refused with a message until the rebuild has
finished, including after an interruption: removing devices and sites,
Expand Down
16 changes: 8 additions & 8 deletions cli/upgrade_partitioning.php
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,9 @@
/*
* cli/upgrade_partitioning.php
*
* Checks how mac_track_ports (MAC addresses) and mac_track_ips (IP
* addresses) are partitioned against the selected scheme and, when they have
* drifted from it or the scheme is being changed, rebuilds them one after the
* Checks how mac_track_ports (MAC addresses), mac_track_ips (IP
* addresses) and mac_track_dot1x are partitioned against the selected scheme
* and, when they have drifted from it or the scheme is changing, rebuilds them one after the
* other:
*
* 1. the table is renamed to <table>_rebuild;
Expand Down Expand Up @@ -200,7 +200,7 @@ function display_help(): void {
mactrack_upart_out('');
mactrack_upart_out('usage: upgrade_partitioning.php [--partition=daily|weekly|disabled] [--debug] [--version] [--help]');
mactrack_upart_out('');
mactrack_upart_out('Checks how the MAC address (mac_track_ports) and IP address (mac_track_ips)');
mactrack_upart_out('Checks the MAC address (mac_track_ports), IP address (mac_track_ips) and 802.1x (mac_track_dot1x)');
mactrack_upart_out('tables are partitioned. Without --partition, the Port and IP Address Partition Size setting');
mactrack_upart_out('is the scheme; a table that already follows it is left alone and reported.');
mactrack_upart_out('A table that has drifted from the scheme, or is not partitioned as asked, is');
Expand Down Expand Up @@ -330,7 +330,7 @@ function mactrack_upart_upgrade_active(): bool {
}

/**
* Checks both tables against the scheme and rebuilds the ones that need it.
* Checks the history tables against the scheme and rebuilds the ones that need it.
*
* @param string $scheme day, week or disabled.
* @param bool $explicit True when --partition chose it, so it is saved as
Expand Down Expand Up @@ -436,7 +436,7 @@ function mactrack_upart_assess_all(string $scheme): array {
* follows the scheme, rebuild or resume when it must be rebuilt, and missing
* when the table does not exist.
*
* @param string $table mac_track_ports or mac_track_ips.
* @param string $table History table.
* @param string $scheme day, week or disabled.
*
* @return array{action: string, reason: string}
Expand Down Expand Up @@ -515,7 +515,7 @@ function mactrack_upart_rebuild_all(array $tables, string $scheme, int $pause):

set_config_option('mt_partition_converting', (string) $started);

mactrack_upart_out('WARNING: Scanning is paused and MAC and IP address searching is in maintenance mode until this finishes.');
mactrack_upart_out('WARNING: Scanning is paused and MAC, IP and 802.1x searching is in maintenance mode until this finishes.');
mactrack_upart_out('WARNING: Allow disk space for a second copy of each table and its indexes; this can take hours.');

$total = ['ok' => true, 'reason' => '', 'rows' => 0, 'partitions' => 0, 'granularity' => $scheme];
Expand Down Expand Up @@ -629,7 +629,7 @@ function mactrack_upart_plan(array $epochs, string $scheme, int $retention_days,
* when it already has the scheme, and the copy restarts from the newest scan
* date it holds.
*
* @param string $table mac_track_ports or mac_track_ips.
* @param string $table History table.
* @param string $scheme day, week or disabled.
* @param callable|null $progress Called as ($table, $done, $total, $scan_date, $rows) after each scan date.
* @param int $pause Seconds between DDL retries; the rename waits twice as long.
Expand Down
37 changes: 17 additions & 20 deletions includes/database.php
Original file line number Diff line number Diff line change
Expand Up @@ -1448,6 +1448,19 @@ function mactrack_retry_default_site(): bool {
* false when seeding failed.
*/
function mactrack_setup_database(bool $notify_seed_failure = false): bool {
// Setup also runs without lib/mactrack_functions.php.
include_once(__DIR__ . '/../lib/mactrack_partitions.php');
include_once(__DIR__ . '/../lib/mactrack_partition_ops.php');
include_once(__DIR__ . '/../lib/mactrack_partition_rebuild.php');

$new_history_tables = [];

foreach (MACTRACK_PARTITION_TABLES as $table) {
if (!db_table_exists($table)) {
$new_history_tables[] = $table;
}
}

$data = [];
$data['columns'][] = ['name' => 'row_id', 'unsigned' => true, 'type' => 'bigint(20)', 'NULL' => false, 'auto_increment' => true];
$data['columns'][] = ['name' => 'site_id', 'unsigned' => true, 'type' => 'int(10)', 'NULL' => false, 'default' => '0'];
Expand Down Expand Up @@ -1730,18 +1743,8 @@ function mactrack_setup_database(bool $notify_seed_failure = false): bool {
$data['keys'][] = ['name' => 'scan_date', 'columns' => 'scan_date'];
$data['type'] = 'InnoDB';
$data['comment'] = '';
$ips_existed = db_table_exists('mac_track_ips');
api_plugin_db_table_create('mactrack', 'mac_track_ips', $data);

if (!$ips_existed) {
// Setup loads this file without lib/mactrack_functions.php.
include_once(__DIR__ . '/../lib/mactrack_partitions.php');
include_once(__DIR__ . '/../lib/mactrack_partition_ops.php');
include_once(__DIR__ . '/../lib/mactrack_partition_rebuild.php');

mactrack_partition_fresh_install(null, 'mac_track_ips');
}

$data = [];
$data['columns'][] = ['name' => 'mac_address', 'type' => 'varchar(20)', 'NULL' => false, 'default' => ''];
$data['columns'][] = ['name' => 'mac_id', 'unsigned' => true, 'type' => 'int(10)', 'NULL' => false, 'auto_increment' => true];
Expand Down Expand Up @@ -1814,18 +1817,8 @@ function mactrack_setup_database(bool $notify_seed_failure = false): bool {
$data['keys'][] = ['name' => 'authorized', 'columns' => 'authorized'];
$data['type'] = 'InnoDB';
$data['comment'] = 'Database for Tracking Device MACs';
$ports_existed = db_table_exists('mac_track_ports');
api_plugin_db_table_create('mactrack', 'mac_track_ports', $data);

if (!$ports_existed) {
// Setup loads this file without lib/mactrack_functions.php.
include_once(__DIR__ . '/../lib/mactrack_partitions.php');
include_once(__DIR__ . '/../lib/mactrack_partition_ops.php');
include_once(__DIR__ . '/../lib/mactrack_partition_rebuild.php');

mactrack_partition_fresh_install();
}

$data = [];
$data['columns'][] = ['name' => 'device_id', 'type' => 'int(11)', 'NULL' => false, 'default' => '0'];
$data['columns'][] = ['name' => 'process_id', 'unsigned' => true, 'type' => 'int(10)', 'NULL' => true];
Expand Down Expand Up @@ -2131,6 +2124,10 @@ function mactrack_setup_database(bool $notify_seed_failure = false): bool {
VALUES ('92xx Switch-default','Cisco','1','*CAT9K_LITE_IOSXE*','','get_IOS_dot1dTpFdbEntry_ports','get_standard_arp_table','0','',0,0,'on')");
}

foreach ($new_history_tables as $table) {
mactrack_partition_fresh_install(null, $table);
}

// Seed only after the complete schema and built-in device types exist.
return mactrack_ensure_default_site($notify_seed_failure, $notify_seed_failure);
}
4 changes: 2 additions & 2 deletions lib/mactrack_activity.php
Original file line number Diff line number Diff line change
Expand Up @@ -266,7 +266,7 @@ function mactrack_activity_state(?int $now = null): array {
if ($converting !== '' || mactrack_partition_rebuild_pending() !== []) {
return array_merge($state, [
'state' => 'converting',
'text' => __('Repartitioning the port and IP tables', 'mactrack'),
'text' => __('Repartitioning the port, IP and 802.1x tables', 'mactrack'),
'since' => $converting === '' ? null : (int) $converting,
]);
}
Expand Down Expand Up @@ -326,7 +326,7 @@ function mactrack_activity_deferred(int $now): ?array {

$labels = [
'upgrade' => __('A database upgrade is running', 'mactrack'),
'partition_rebuild' => __('The port and IP table partitioning has not finished', 'mactrack'),
'partition_rebuild' => __('The port, IP and 802.1x table partitioning has not finished', 'mactrack'),
];
$reason = $labels[$note['reason'] ?? ''] ?? __('The aggregated ports rebuild has not finished', 'mactrack');

Expand Down
10 changes: 7 additions & 3 deletions lib/mactrack_functions.php
Original file line number Diff line number Diff line change
Expand Up @@ -3414,12 +3414,16 @@ function mactrack_db_maint_run(array &$steps, ?int $now = null): void {
// daily DELETE frees.

// The ARP translations follow the IP retention and the 802.1x sessions
// the port retention; nothing else ever removed them.
// the port retention.
db_execute_prepared('DELETE FROM mac_track_arp
WHERE scan_date < CURDATE() - INTERVAL ? DAY',
[$ip_retention]);

if ($days > 0) {
if (mactrack_partition_state('mac_track_dot1x')['layout']['state'] === 'range') {
$result = mactrack_partition_maintain((int) $days, $now, 'mac_track_dot1x');

mactrack_debug('Dropped ' . count($result['dropped']) . ' and created ' . count($result['created']) . ' 802.1x partitions' . ($result['reason'] !== '' ? " ({$result['reason']})" : '') . '.');
} elseif ($days > 0) {
mactrack_delete_batched('DELETE FROM mac_track_dot1x
WHERE scan_date < ?
ORDER BY scan_date',
Expand Down Expand Up @@ -6075,7 +6079,7 @@ function mactrack_utilities_post_actions(): array {
'mactrack_refresh_oui_database' => __('Are you sure you want to download and import the IEEE Vendor MAC/OUI database now?', 'mactrack'),
'mactrack_utilities_rebuild_scan_dates' => __('Are you sure you want to rebuild the scan date list from the stored Port to MAC to IP results now?', 'mactrack'),
'mactrack_utilities_optimize_ports' => __('Rebuild and analyze the partition containing the latest port observation, which is the current week with weekly partitions. An unpartitioned table is optimized in full. Scheduled scans are skipped while this runs. Continue?', 'mactrack'),
'mactrack_utilities_convert_partitions' => __('Run this maintenance off hours. The rebuild copies the full history of the port (MAC address) and IP address tables one after the other and can take hours. Scheduled scans are skipped, database maintenance waits, and MAC and IP address searching shows a maintenance message until it finishes. Allow free disk space for a second copy of each table and its indexes. Continue?', 'mactrack'),
'mactrack_utilities_convert_partitions' => __('Run this maintenance off hours. The rebuild copies the full history of the port (MAC address), IP address and 802.1x tables one after the other and can take hours. Scheduled scans are skipped, database maintenance waits, and MAC, IP and 802.1x searching shows a maintenance message until it finishes. Allow free disk space for a second copy of each table and its indexes. Continue?', 'mactrack'),
];
}

Expand Down
13 changes: 7 additions & 6 deletions lib/mactrack_partition_ops.php
Original file line number Diff line number Diff line change
Expand Up @@ -635,13 +635,13 @@ function mactrack_partition_load_scan(array $plan): int|false {
* daily partitions, or, for per-scan partitions, daily slots named
* aYYYYMMDD that each scan carves its partition from. Nothing is created
* while dMaxValue holds rows: the next scan's recovery split moves those
* first. The IP table has no scan to move them, so maintenance splits them
* into a recovery partition itself. Other layouts are left to their own
* first. The IP and 802.1x tables have no scan exchange to move them, so
* maintenance splits them into a recovery partition itself. Other layouts are left to their own
* maintenance.
*
* @param int $days Retention in days; below 1 drops nothing.
* @param int|null $now Current time, for tests.
* @param string $table mac_track_ports or mac_track_ips.
* @param string $table History table.
*
* @return array{dropped: string[], created: string[], reason: string}
*/
Expand All @@ -663,7 +663,8 @@ function mactrack_partition_maintain(int $days, ?int $now = null, string $table
}
}

mactrack_partition_status([($table === 'mac_track_ports' ? 'maintenance' : 'maintenance_ips') => $result + ['time' => time()]]);
$status_key = ['mac_track_ports' => 'maintenance', 'mac_track_ips' => 'maintenance_ips', 'mac_track_dot1x' => 'maintenance_dot1x'][$table];
mactrack_partition_status([$status_key => $result + ['time' => time()]]);

return $result;
}
Expand All @@ -675,7 +676,7 @@ function mactrack_partition_maintain(int $days, ?int $now = null, string $table
* @param array{mariadb: bool, version: string} $server From mactrack_partition_server().
* @param int $days Retention in days.
* @param int $now Current time.
* @param string $table mac_track_ports or mac_track_ips.
* @param string $table History table.
*
* @return array{dropped: string[], created: string[], reason: string}
*/
Expand Down Expand Up @@ -715,7 +716,7 @@ function mactrack_partition_maintain_locked(array $partitions, array $server, in
// Per-scan partitions cannot be created before their scan_date is
// known, so daily slots are created ahead and each scan carves its
// partition off the front of the empty slot that covers it. The IP
// table only has daily and weekly partitions.
// and 802.1x tables only have daily and weekly partitions.
$ahead = $size === 'scan' || ($size === 'hour' && !$ports) ? 'day' : $size;

if (!$ports) {
Expand Down
Loading
Loading