release: Fleet-Ops v0.6.70

.github/workflows/postman.yml

@@ -5,14 +5,9 @@ name: API Contract (Postman)
 # fleetbase/fleetbase. Requires org secrets POSTMAN_API_KEY + _GITHUB_AUTH_TOKEN
 # (inherited); no-ops until POSTMAN_API_KEY is set.
 #
-# Deliberately unpinned. The reusable workflow defaults to booting fleetbase/fleetbase@main
-# against fleetbase/fleetbase-api:latest, so every release is picked up automatically and
-# there is no ref here to remember to bump. Each run records the image digest it actually
-# resolved in its job summary, so a result stays traceable. To reproduce an older run:
-#
-#   with:
-#     fleetbase-ref: v0.7.53
-#     api-image: fleetbase/fleetbase-api:v0.7.53
+# Pin the workflow and host checkout together so branch installation resolves the
+# required published Core release before migrating and exercising this module.
+# The workflow records the API image digest and resolved dependencies for traceability.
 
 on:
   push:
@@ -25,8 +20,9 @@ permissions:
 
 jobs:
   contract:
-    uses: fleetbase/fleetbase/.github/workflows/api-contract.yml@main
+    uses: fleetbase/fleetbase/.github/workflows/api-contract.yml@880c7392c67c93e3a65f01916d7a04036c098935
     with:
+      fleetbase-ref: 880c7392c67c93e3a65f01916d7a04036c098935
       collections: "Fleetbase API"
       build-from-source: false
       # Without this the run tests the version of fleetbase/fleetops-api baked into the

.github/workflows/server.yml

@@ -14,7 +14,7 @@ jobs:
 
     steps:
     - name: Checkout code
-      uses: actions/checkout@v2
+      uses: actions/checkout@v4
 
     - name: Setup PHP
       uses: shivammathur/setup-php@v2
@@ -53,6 +53,7 @@ jobs:
       run: php scripts/coverage-summary.php coverage/clover.xml --fail-under=100
 
     - name: Upload Coverage Baseline
+      if: ${{ always() && hashFiles('coverage/clover.xml') != '' }}
       uses: actions/upload-artifact@v4
       with:
         name: fleetops-coverage-clover

RELEASE.md

@@ -1,23 +1,56 @@
-> v0.6.69 ~ "Fleet-Ops tools for Fleetbase AI"
+> v0.6.70 ~ "Telematics data retention"
 
 ---
 ## What's New
-- **Fleet-Ops works with Fleetbase AI tool calling.** Creating orders from the AI prompt works again with the tool-calling assistant, and the assistant can also propose an optimized waypoint sequence for an order and explain what an order or resource import needs. Every change is a preview card the user confirms.
-- **Fleet-Ops console actions.** The assistant can offer to open Fleet-Ops pages and dialogs, such as **Operations › Orders** and **New Order**, as confirmation cards.
-- **Resource search for the assistant.** A `fleetops_search` tool finds orders, vehicles, drivers, work orders, maintenances, devices, sensors and telematics records by id, name, plate, VIN, email or phone.
+- **Telematics Data settings.** A new Fleet-Ops → Settings → Telematics Data page lets each organization decide how long telemetry is kept: device events, raw event payloads, positions, processed and quarantined delivery envelopes, and sync run diagnostics. Administrators set system-wide defaults from the admin console (Fleet-Ops Config → Telematics Data). A value of 0 keeps rows forever.
+- **Scheduled retention sweep.** `fleetops:prune-telematics-data` runs every fifteen minutes and applies each organization's policy in bounded batches, so a large backlog is drained gradually without stalling ingestion. Defaults: events 30 days, raw payloads stripped after 7 days, positions 90 days, processed deliveries 24 hours, quarantined deliveries 7 days, sync runs 7 days.
+- **Storage usage and on-demand cleanup.** The settings page shows rows, oldest row and estimated size per table for your organization, and "Run cleanup now" queues an immediate retention run.
 
 ---
 ## Fixes
-- AI resource search no longer fails on every call with `Unknown column 'sensor_type'`, and database errors are no longer passed to the model.
+- Device events no longer store the raw provider unit twice. `payload` keeps the raw unit; `meta` holds only the normalized block (speed, heading, odometer, ignition, fuel level, timing). New event rows are roughly half the size.
+- Telemetry-driven saves (device, event, position, vehicle, sensors) no longer write to the activity log on every poll. Organizations that want them can enable "Log telemetry activity" in Telematics Data settings. Manual edits are logged as before.
+- Delivery inbox and sync run retention moved from the drain command into the retention sweep; `fleetops:drain-telematic-inbox` now only recovers deliveries and finishes interrupted runs.
+
+---
+## Upgrade notes
+- New indexes on `device_events (company_uuid, created_at)`, `positions (company_uuid, created_at)` and `telematic_sync_runs (telematic_uuid, updated_at)`; on very large tables run `php artisan migrate` in a maintenance window.
+- Retention is enabled by default. Existing rows older than the defaults are removed over successive runs after upgrading; set the values to 0 before upgrading if you need to keep history indefinitely.
+- Deleting rows frees space inside the database files, not on disk. Run `OPTIMIZE TABLE` off-peak to return space to the operating system.
+- Run `php artisan fleetbase:create-permissions` to register the `telematics-settings` permission and the Telematics Settings Manager policy and role.
+
+---
+## Testing
+- Backend tests cover the retention policy layering and clamping, the prune command (per-company policies, compaction, inbox tables via connections, orphans, batch caps, dry runs, filters, locking), the settings and storage usage endpoints, the index migrations, the on-demand job, and the meta and activity-log changes to ingestion.
+- Ember tests cover the settings route guard, the settings controller, and the admin defaults component.
+> v0.6.70 ~ "Order reporting and a cleaner default dashboard"
+
+---
+## What's New
+- **Report on orders, their items and tracking.** The report builder can answer questions like "Which products sold the most this month?" and "What did orders total this month?".
+  - Orders expose their ID and Internal ID, and relationships for tracking (tracking number, region, latest status), Order Config, Customer Vendor, Created By, Purchase Rate › Service Quote and Payload Return.
+  - `Payload` exposes its items with name, SKU, price, dimensions, metadata and destination. Select an item column for one row per item, or group by one to summarise per product.
+  - Summary columns count orders distinctly, so they stay correct when item rows are selected. Distance, duration and transaction amount have totals and averages.
+  - `meta` has no fixed shape, so no column assumes a key in it. Read keys with a computed column, for example `CAST(JSON_UNQUOTE(JSON_EXTRACT(payload.entities.meta, '$.quantity')) AS DECIMAL(15,2))`.
+  - Column labels drop the table name ("Type", not "Order Type"). Distance and duration are labelled in meters and seconds, and money columns are marked "(minor units)".
+- **A cleaner default dashboard.** Fleet-Ops widgets declare their place on the default dashboard: Radar first, then Active Orders and Drivers Online, a full-width Live Fleet map, and Revenue Trend, Top Drivers and Maintenance side by side.
+- **Live Fleet widget cards match the live map.** Clicking a driver or vehicle on the widget map shows the same card as **Operations › Live Map**.
+
+---
+## Fixes
+- Custom field values on fuel reports and service areas are saved. They were dropped because the save hooks read the snake_case payload root, while the console sends `fuelReport` and `serviceArea`.
+- The report schema no longer references 13 columns and join keys that don't exist: driver name, email and phone and the driver's vehicle, the vehicle's driver, and the fuel report cost, odometer and date. Transaction summaries sum `transaction.amount`.
+- Every report table, and the item join, leaves soft-deleted rows out on core-api v1.6.64 and later.
 
 ---
 ## Dependencies
-- `@fleetbase/fleetops-data` upgraded to `^0.2.2`, which adds the read-only login state (`is_staff_linked`, `login_status`) to the driver and contact models.
-- `@fleetbase/ember-ui` upgraded to `^0.4.3`, which provides the `btn-auth` style the Track Order button on the login page uses.
+- Order items, JSON totals and soft deletes in reports need `fleetbase/core-api` v1.6.64. The schema still registers on older versions.
+- The report builder and the default dashboard order ship in `@fleetbase/ember-ui` v0.4.4.
 
 ---
 ## Testing
-- Unit tests cover the new AI tools and console commands, and the AI capability registration.
+- Every declared report column and join key was checked against the database schema, and order reports were run against real storefront orders on MySQL in strict mode.
+- Tests cover the report schema at 100% line coverage, the Live Fleet map cards and the default dashboard order.
 
 ---
 ## Need help?

addon/components/admin/telematics-settings.hbs

@@ -0,0 +1,143 @@
+<div class="space-y-6">
+    <InfoBlock @type="info" @icon="circle-info" class="mb-4">
+        Configure telematics retention policy and activity logging for all organizations. Organizations can choose device event and position history retention within these limits from Fleet-Ops &rsaquo; Settings &rsaquo; Telematics.
+        A maximum takes precedence over organization preferences, inherited defaults and requests to keep history forever. Reducing a maximum can delete existing history at the next cleanup.
+    </InfoBlock>
+
+    <ContentPanel @title="System Device Events Retention Policy" @open={{true}} @wrapperClass="bordered-classic">
+        <div class="grid grid-cols-1 md:grid-cols-2 gap-4">
+            <InputGroup @name="Default Device Event Retention (days)" @helpText="Retention for organizations that inherit the system default. 0 keeps history forever, subject to the maximum below.">
+                <Input @type="number" @value={{this.eventRetentionDays}} min="0" max="3650" class="form-input w-full" />
+            </InputGroup>
+            <InputGroup @name="Maximum Device Event Retention (days)" @helpText="The longest any organization can keep device events. 0 sets no maximum and allows organizations to keep history forever.">
+                <Input @type="number" @value={{this.maxEventRetentionDays}} min="0" max="3650" class="form-input w-full" />
+            </InputGroup>
+        </div>
+    </ContentPanel>
+
+    <ContentPanel @title="System Position History Retention Policy" @open={{true}} @wrapperClass="bordered-classic">
+        <div class="grid grid-cols-1 md:grid-cols-2 gap-4">
+            <InputGroup @name="Default Position History Retention (days)" @helpText="Retention for organizations that inherit the system default. 0 keeps history forever, subject to the maximum below.">
+                <Input @type="number" @value={{this.positionRetentionDays}} min="0" max="3650" class="form-input w-full" />
+            </InputGroup>
+            <InputGroup @name="Maximum Position History Retention (days)" @helpText="The longest any organization can keep vehicle and trailer positions. 0 sets no maximum and allows organizations to keep history forever.">
+                <Input @type="number" @value={{this.maxPositionRetentionDays}} min="0" max="3650" class="form-input w-full" />
+            </InputGroup>
+        </div>
+    </ContentPanel>
+
+    <ContentPanel @title="System Device Event Raw Payload Retention" @open={{true}} @wrapperClass="bordered-classic">
+        <InputGroup @name="Device Event Raw Payload Retention (days)" @helpText="Remove raw provider payloads from retained device events across all organizations after this many days. 0 disables compaction. This does not change how long the event history itself is kept.">
+            <Input @type="number" @value={{this.eventCompactAfterDays}} min="0" max="3650" class="form-input w-full" />
+        </InputGroup>
+    </ContentPanel>
+
+    <ContentPanel @title="System Delivery Inbox and Sync Run Retention Policy" @open={{true}} @wrapperClass="bordered-classic">
+        <div class="grid grid-cols-1 md:grid-cols-2 gap-4">
+            <InputGroup @name="Processed Delivery Retention (hours)" @helpText="Delete processed delivery envelopes across all organizations after this many hours. 0 keeps them forever.">
+                <Input @type="number" @value={{this.processedRetentionHours}} min="0" max="720" class="form-input w-full" />
+            </InputGroup>
+            <InputGroup @name="Quarantined Delivery Retention (days)" @helpText="Delete quarantined delivery envelopes across all organizations after this many days. 0 keeps them forever.">
+                <Input @type="number" @value={{this.quarantineRetentionDays}} min="0" max="365" class="form-input w-full" />
+            </InputGroup>
+            <InputGroup @name="Sync Run Diagnostics Retention (days)" @helpText="Delete finished sync run diagnostics across all organizations after this many days. 0 keeps them forever.">
+                <Input @type="number" @value={{this.syncRunRetentionDays}} min="0" max="365" class="form-input w-full" />
+            </InputGroup>
+        </div>
+    </ContentPanel>
+
+    <ContentPanel @title="System Telemetry Activity Logging" @open={{true}} @wrapperClass="bordered-classic">
+        <Toggle
+            @label="Log Telemetry Activity"
+            @helpText="Write telemetry-driven device, event and vehicle saves to the activity log for all organizations. Every poll can add several activity rows per device."
+            @isToggled={{this.logTelemetryActivity}}
+            @onToggle={{fn (mut this.logTelemetryActivity)}}
+        />
+    </ContentPanel>
+
+    <div class="flex justify-end">
+        <Button
+            @type="primary"
+            @size="sm"
+            @icon="save"
+            @text="Save System Policy"
+            @onClick={{perform this.saveSettings}}
+            @disabled={{or this.saveSettings.isRunning this.loadSettings.isRunning (not this.settingsLoaded)}}
+            @isLoading={{or this.saveSettings.isRunning this.loadSettings.isRunning}}
+        />
+    </div>
+
+    <ContentPanel @title={{t "settings.telematics.storage-usage"}} @open={{true}} @wrapperClass="bordered-classic">
+        <p class="mb-3 text-sm text-gray-600 dark:text-gray-300">
+            Storage usage includes telematics data across all organizations. Cleanup applies system-wide, using the saved system policy and each organization's history preferences within that policy.
+        </p>
+        <div class="flex items-center justify-end gap-2 mb-3">
+            <Button
+                @size="xs"
+                @icon="rotate"
+                @text={{t "settings.telematics.refresh"}}
+                @onClick={{perform this.loadUsage}}
+                @disabled={{this.loadUsage.isRunning}}
+                @isLoading={{this.loadUsage.isRunning}}
+            />
+            <Button
+                @type="danger"
+                @size="xs"
+                @icon="broom"
+                @text={{t "settings.telematics.run-cleanup"}}
+                @onClick={{this.confirmRunCleanup}}
+                @disabled={{or this.runCleanup.isRunning this.loadSettings.isRunning this.saveSettings.isRunning (not this.settingsLoaded)}}
+                @isLoading={{this.runCleanup.isRunning}}
+            />
+        </div>
+        {{#if this.usageRows.length}}
+            <div class="overflow-x-auto">
+                <table class="w-full text-sm">
+                    <thead>
+                        <tr class="text-left text-xs uppercase text-gray-500 dark:text-gray-400">
+                            <th class="py-2 pr-3">{{t "settings.telematics.table"}}</th>
+                            <th class="py-2 pr-3 text-right">{{t "settings.telematics.rows"}}</th>
+                            <th class="py-2 pr-3 text-right">{{t "settings.telematics.estimated-size"}}</th>
+                            <th class="py-2 pl-6 text-right whitespace-nowrap">{{t "settings.telematics.oldest"}}</th>
+                        </tr>
+                    </thead>
+                    <tbody class="divide-y divide-gray-100 dark:divide-gray-700">
+                        {{#each this.usageRows as |row|}}
+                            <tr class="text-gray-800 dark:text-gray-100">
+                                <td class="py-2 pr-3">
+                                    <div class="font-medium">{{row.label}}</div>
+                                    {{#if row.raw_payload_rows}}
+                                        <div class="text-xs text-gray-500 dark:text-gray-400">
+                                            {{t "settings.telematics.raw-payload-rows" count=(format-number row.raw_payload_rows) compactable=(format-number row.compactable_rows)}}
+                                        </div>
+                                    {{/if}}
+                                </td>
+                                <td class="py-2 pr-3 text-right">
+                                    {{#if (eq row.rows null)}}
+                                        {{t "settings.telematics.not-available"}}
+                                    {{else if row.rows_estimated}}
+                                        {{t "settings.telematics.estimated-rows" count=(format-number row.rows)}}
+                                    {{else}}
+                                        {{format-number row.rows}}
+                                    {{/if}}
+                                </td>
+                                <td class="py-2 pr-3 text-right">{{#if (or row.estimated_bytes (eq row.estimated_bytes 0))}}{{format-bytes row.estimated_bytes}}{{else}}{{t "settings.telematics.not-available"}}{{/if}}</td>
+                                <td class="py-2 pl-6 text-right text-xs whitespace-nowrap">{{#if row.oldest}}{{row.oldest}}{{else}}{{t "settings.telematics.not-available"}}{{/if}}</td>
+                            </tr>
+                        {{/each}}
+                    </tbody>
+                </table>
+            </div>
+            <p class="mt-3 text-xs text-gray-500 dark:text-gray-400">{{t "settings.telematics.space-reclaim-note"}}</p>
+        {{else if this.loadUsage.isRunning}}
+            <div class="flex items-center justify-center py-8">
+                <Spinner />
+            </div>
+        {{else}}
+            <div class="text-center py-8 text-sm text-gray-400 dark:text-gray-500">
+                {{t "settings.telematics.usage-unavailable"}}
+            </div>
+        {{/if}}
+    </ContentPanel>
+    <Spacer @height="200px" />
+</div>