Conversation
Deep-dive concept page (concepts/deep-dive/expire-rows.md) for passthrough materialized views with EXPIRE ROWS: all modes, worked examples, read-filter/cleanup mechanics, NULL/ties/monotonicity semantics, and limitations. Adds an ALTER MATERIALIZED VIEW SET EXPIRE reference page, an EXPIRE ROWS section in CREATE MATERIALIZED VIEW, and sidebar entries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
nwoolmer
added
documentation
|
🚀 Build success! Latest successful preview: https://preview-468--questdb-documentation.netlify.app/docs/ Commit SHA: 98bcfe6
|
Corrections and additions ported from the core repo's in-repo design doc (docs/row-expiry.md, now removed there): - Aggregating views are ACCEPTED with a logged advisory, not rejected (the previously-cited error message does not exist); reworded the concept note, Requirements, the ALTER behavior and errors tables, and the CREATE section accordingly. - materialized_views() exposes the policy as expire_clause, not expire_predicate. - Monotonicity: list the exact clock shapes the cleanup job proves monotonic (bare clock, clock minus non-negative constant, fixed-unit look-back dateadd) and that calendar units, look-forward offsets, compound arithmetic, and window predicates skip cleanup; the job skips rather than risks deleting rows. - No-policied-chains rule: CREATE rejects a defining query reading a policied view (base or join); SET EXPIRE is rejected with dependent views; corresponding errors added to the ALTER page. - Reserved __qdb_re_keep column name; no line comments inside the clause; CLEANUP EVERY strict <number><unit> grammar (s/m/h/d/w). - Kill switch is read at startup (restart required); failing sweeps back off from 1s up to a 10-minute cap. - Parquet side effect: compacting a partially-expired Parquet partition rewrites it as native storage until re-conversion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Passthrough materialized views only appeared in the docs as a
prerequisite of EXPIRE ROWS. The single sentence describing which
queries qualify lived in the EXPIRE ROWS concept page, and every
example on every page was literally "SELECT * FROM base", so a reader
who wanted a maintained subset of a table without a retention policy
had no reason to find any of it.
Three statements also still said a materialized view query has to
aggregate, which is no longer true:
- "Materialized views require a `SAMPLE BY` or time-based `GROUP BY`
query." in the concepts page
- "Must use `SAMPLE BY` or `GROUP BY` with a designated timestamp
column" in the same page's Technical reference
- "Query must aggregate" in the CREATE MATERIALIZED VIEW rules table
Give the shape its own section in the general materialized views
concept page: what a passthrough view is, when to reach for one (a
narrowed replica of a big table, or a target for EXPIRE ROWS), what it
inherits from the base table, and which queries qualify.
The inherited-properties part records something documented nowhere
before: a passthrough view inherits the base table's symbol indexes,
under whatever alias the projection gives the column, so an indexed
lookup on the view costs what it costs on the base. An aggregating view
never inherits an index, because its rows are not base rows.
The query table lists shapes verified by running them, rather than read
off isPassthrough(). A column subset, a column alias and a row-local
expression such as "price * amount AS notional" are all passthrough. A
window function is rejected, but not by either check inside
isPassthrough() -- a separate guard reports
window function on base table is not supported for materialized views
Dropping the designated timestamp, including by an ORDER BY on another
column, is rejected with
materialized view query is required to have designated timestamp
and REFRESH PERIOD is rejected with
PERIOD is not supported for non-aggregating (passthrough) materialized views
while REFRESH MANUAL and REFRESH EVERY are both accepted.
Add a section to the EXPIRE ROWS page on choosing between a WHERE
clause in the view's query and an EXPIRE ROWS WHEN predicate. For a
deterministic predicate that judges each row on its own the two keep
the same rows, and nothing said how to pick between them. A WHERE
clause is cheaper on storage, on reads and on writes, because the row
is never written. An EXPIRE ROWS policy can be retuned with ALTER
instead of forcing a drop and re-create, which re-reads the base table
and can lose rows the base no longer holds. A policied view also cannot
serve as another view's base.
Also correct the claim that JOIN enrichment fails because SAMPLE BY is
mandatory. A passthrough view does keep raw rows; it just has to read a
single table, so enrichment stays unavailable for a different reason.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A predicate that looks only at the row's own values, such as EXPIRE ROWS WHEN amount < 1.5 selects the same surviving rows as writing the opposite test in the view's own query: SELECT * FROM trades WHERE amount >= 1.5 The `WHERE` version is cheaper everywhere. The rows it excludes are never copied into the view, so they cost no disk, no read-time filtering, and no rewriting of partitions by the cleanup job. The `EXPIRE ROWS` version writes those rows, hides them on every read, and reclaims them later. The docs presented the two as a roughly even choice and left the one thing only `EXPIRE ROWS WHEN` can do — a cutoff that moves with the clock — as a footnote at the end of the comparison. Someone reading quickly could reasonably conclude that any filter is fine to write as a policy. The comparison section now opens with the rule instead: use `EXPIRE ROWS WHEN` for predicates that involve wall-clock time, and put a predicate that depends only on the row's values in the `WHERE` clause. A rolling window genuinely has no `WHERE` equivalent, because a view's defining query rejects non-deterministic functions and will not accept `now()`. The argument for keeping a fixed, hand-advanced cutoff in a policy is still true and still here, moved into its own subsection so it reads as the exception rather than an equal option. The section keeps its `#where-filter-or-expire-rows` anchor, which `materialized-views.md` links to. Two other notes were added while rewriting. The two forms are not exact opposites where `NULL`s are concerned: `WHERE` keeps a row only when the test is `TRUE`, while `EXPIRE ROWS WHEN` expires a row only when the test is `TRUE`, so a `NULL` amount is dropped by the first and kept by the second. And the worked example that expires small trades now says up front that it is written that way to make the kept rows easy to read off the sample data, not because that is where such a filter belongs. The same guidance was added to the two SQL reference pages, so a reader who lands on one of them without reading the concept page still sees it. The `ALTER MATERIALIZED VIEW SET EXPIRE` page previously opened its examples with `WHEN amount < 1.5`, which demonstrated exactly the shape being discouraged; that example is now a rolling seven-day window. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The passthrough views section gave two reasons to reach for one, a
narrowed replica and a target for EXPIRE ROWS, as bullet points with no
SQL under them. A reader who wanted either had nothing to copy. Add a
"Use cases" section with three worked examples, each opening with a
one-sentence scenario and naming the domain it comes from, so the
section covers sensor telemetry as well as capital markets.
The examples are a current-reading-per-sensor view using KEEP LATEST, a
live options chain using a WHEN predicate, and the largest trades per
symbol using KEEP 10 HIGHEST. Every statement was run against a live
server before being written down.
Two behaviors are recorded here that were not documented anywhere else.
First, the cleanup job reclaims disk only when the cutoff names the
view's designated timestamp. The options example compares a contract's
expiry column against now(), and materialized_views() reports
options_live expiry < now() FILTER_ONLY
options_ctl ts < dateadd('d', -7, now()) FILTER_AND_RECLAIM
for two views over the same base table, so the column is what decides
it, not the shape of the predicate. Expired contracts stop showing up
in queries straight away either way; only the disk behavior differs.
Second, when two rows tie at the Nth place under KEEP N, the newer row
survives.
An order book example was written first and then dropped. KEEP N
HIGHEST ranks over everything the view holds rather than over a recent
window, so a view of the ten highest bids per symbol fills up with the
highest prices ever seen. With one bid from the previous day at 200.00
and a live book around 100, the stale bid outranked every current one
and the live bids were expired out of the view. An order book is state
that gets superseded, and a materialized view can only add rows, so no
policy can remove an order that was later cancelled. The trades example
replaces it because a trade is never retracted, which makes ranking
over all history the intended meaning rather than a defect. The closing
paragraph of that example now points at KEEP LATEST for values that do
get superseded, to save the next reader the same detour.
Also drop the list of policy shapes from the second bullet, since the
examples below it now demonstrate each one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CREATE and ALTER now refuse a scalar WHEN threshold that is constant at definition time and evaluates to NULL, since such a policy expires nothing. Add a Semantics section covering the explicit CAST(NULL AS TIMESTAMP) and the arithmetic spellings that overflow onto the reserved NULL value, and note that a clock-based threshold is evaluated per read and so cannot be checked in advance. Add the error to the SET EXPIRE error table, and a pointer to the new section from the CREATE MATERIALIZED VIEW reference. Also steer the rolling-window example toward dateadd. It and now() - 86400000000 retain the same rows, but only the dateadd form reduces to a timestamp range the scan can use to skip partitions; the arithmetic form is evaluated row by row across the whole view on every read. Both reclaim disk, so the difference is read cost alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Verdict: approve with nits All four blocking items are fixed and verified. yarn build passes clean. Resolved:
One thing I checked because the new trades_mirror example omits it: passthrough views inherit the base table's partitioning when PARTITION BY is absent (CreateMatViewOperationImpl:989-996), and every MatViewExpireRowsTest case creates them that way. So all the bare CREATE MATERIALIZED VIEW ... AS (SELECT * FROM trades) EXPIRE ROWS ... examples are valid as written. Still open, all non-blocking and your call — the Minors from last round are unchanged (sidebars.js:346 alphabetical order, "the top-N form keeps exactly one row per group" at expire-rows.md:46, the truncated KEEP 2 HIGHEST price ... at :624, the missing backlink in alter-mat-view-set-ttl.md, "summarising" at materialized-views.md:18, the meta.md whitespace churn, 'BTC' vs demo BTC-USDT, description length). One new, introduced by the rename: materialized-views.md:214 is now the only surviving bare ts in the changed pages — LATEST ON ts PARTITION BY sensor_id, on a sensor_readings table whose schema the page never gives. It read as fine when everything else used ts; now it reads as a stray. Either name the column timestamp or give sensor_readings a one-line schema. The two claims I couldn't verify last time are unchanged and still worth a confirm from you: the Parquet-compaction-reverts-to-native note (expire-rows.md:678-681) and the Enterprise primary-only cleanup claim (:435-437). |
Explain the LIMIT restriction on query branches that read the base table, including nested queries, in the CREATE MATERIALIZED VIEW reference and documentation changelog. Describe the effect on recreating existing views and replaying SHOW CREATE output. Explain that applying the limit when querying the view changes which rows it stores. Limited subqueries over other tables remain allowed. Validate both files with the MDX parser, check the changelog section link, and verify the diff has no whitespace errors.
Replace the obsolete SET EXPIRE dependency rejection rule with refresh- time conflict detection for existing materialized and live views. Explain delayed invalidation, completion against earlier snapshots, and why source expiry does not retroactively filter stored dependent rows. Document FULL refresh and live-view recreation as recovery steps after resolving a source conflict. Correct the general FULL-refresh visibility and failure contract: rebuilding starts with a truncate and does not restore previous contents after a later failure. Validate the documentation with a successful Docusaurus production build.
| | --------------------- | --------------------------------------------------- | ------------------------------------------------------------------- | --------------------- | | ||
| | Per-row predicate | Rows for which the predicate is **not** `TRUE` | `EXPIRE ROWS WHEN predicate` | Yes, when monotonic | | ||
| | Keep latest | The latest row per key (current state per key) | `EXPIRE ROWS KEEP LATEST [ON timestamp] PARTITION BY cols` | No (read filter only) | | ||
| | Keep highest / lowest | Rows tied at the group max / min of a column | `EXPIRE ROWS KEEP HIGHEST\|LOWEST col [PARTITION BY cols]` | No (read filter only) | |
There was a problem hiding this comment.
I know this is not just docs, but I find it weird that LATEST is ON and HIGHEST/LOWEST do not use ON. A bit of a UX friction
There was a problem hiding this comment.
LATEST ON is pre-existing syntax, and this actually gets rewritten to the existing query. So it makes sense to keep it for familiarity, although I don't know why it's even there -- you MUST name the designated timestamp, it's 100% redundant.
Given that, would you then add ON to KEEP HIGHEST/LOWEST?
There was a problem hiding this comment.
We actually had some versions in which ON was optional and you could just do PARTITION BY, but that works only for some queries and I think it is deprecated, so we always want to use the ON. Not sure why. But I would like the consistency in the ROWS KEEPS. It feels wrong that one is ROWS KEEP LATEST ON Column, and the other ROWS KEEP HIGHEST Column
| ``` | ||
|
|
||
| A `WHEN` predicate is for rules that move with **wall-clock time**, such as a | ||
| rolling `timestamp < dateadd('d', -7, now())` window. The defining query cannot |
There was a problem hiding this comment.
I'd rather use TICK syntax here
| @@ -12,6 +12,11 @@ their results at query time, materialized views persist their data to disk, | |||
| making them particularly efficient for expensive aggregate queries that are run | |||
There was a problem hiding this comment.
General comment for this file, not for this line.
In the performance section of materialized views, we should probably mention the extra queries that are executed when the views have expiring rows?
There was a problem hiding this comment.
Also, we have a whole section on optimizing for latest on, which EXPIRING VIEWS completely made obsolete, right?
There was a problem hiding this comment.
Added performance notes in 5eac2ec
As for optimizing LATEST ON, unfortunately row expiry doesn't help here because it still keeps the entire history, and only adds more clauses to the user's query. The feature actually deletes data only for a certain clas of EXPIRE WHEN predicates, those that provably expire all rows before a cutoff point.
There was a problem hiding this comment.
Then we can probably add it there, as when I saw we can use EXPIRE ROWS KEEP LATEST ON ts, I thought I could just use that instead of going through the hoops of an aggregating matview. We should explain probably the tradeoff and show the current way of doing it efficiently
| See the [Expiring rows](/docs/concepts/expire-rows/) concept page for | ||
| all modes, worked examples, and semantics (NULLs, ties, monotonicity). | ||
|
|
||
| ## Complete example |
There was a problem hiding this comment.
The complete example is only for aggregating views. We should probably say complete examples and add an example of each?
Or just remove the complete example
There was a problem hiding this comment.
Also, the links at the bottom of this page link to alter view TTL, but not to alter view EXPIRE ROWS
3 participants
Deep-dive concept page (concepts/deep-dive/expire-rows.md) for passthrough materialized views with EXPIRE ROWS: all modes, worked examples, read-filter/cleanup mechanics, NULL/ties/monotonicity semantics, and limitations. Adds an ALTER MATERIALIZED VIEW SET EXPIRE reference page, an EXPIRE ROWS section in CREATE MATERIALIZED VIEW, and sidebar entries.