diff --git a/CHANGELOG.md b/CHANGELOG.md index 072b0a70..04d50105 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,18 @@ ## Unreleased +- Changed (runtime contract, within the promise): a value-semantics copy is + released before the kernel returns when its last variable dies and no + other variable may share the buffer, instead of living until return. A + copy made every iteration of a loop (`b = a; a[i] = x; ... b[0]`) no + longer keeps one buffer per iteration under `max_owned_bytes`: the loop + holds at most two. `alloc/2` buffers, bitstring accumulators and aligned + input copies still live until return, so Elixir mode's budget still + raises at the same `alloc/2` as native code. The fast-path report notes + each release (`:owned`, `:released`), and the generated C shows them as + `defk_disown` calls. Results are unchanged; a kernel that raised "out of + memory" at the budget only because of such copies now runs to its result + (the budget's numbers are outside the promise, `docs/guides/stability.md`). - Changed (runtime, no promise changes): on Apple platforms the clock that measures a yielding kernel's slices reads `mach_absolute_time` instead of the ERTS monotonic clock. A yielding kernel's call cost about 20 ns less diff --git a/docs/adr/0012-early-release-of-dead-copies.md b/docs/adr/0012-early-release-of-dead-copies.md new file mode 100644 index 00000000..787e36eb --- /dev/null +++ b/docs/adr/0012-early-release-of-dead-copies.md @@ -0,0 +1,100 @@ +# ADR-0012: A dead copy is released early; an alloc lives until return + +- Date: 2026-10-22 +- Status: accepted (2026-10-22: the maintainer chose "copies only" from + the design proposal on the ticket) +- Source: ticket `defk-c1e` in epic `defk-w8r`, follow-up to `defk-4b2` + (the owned-buffer budget) + +## Context + +Every buffer a kernel call owns (`alloc/2`, a value-semantics copy, a +bitstring accumulator, an aligned copy of a misaligned input) is a slot in +`ctx->owned` and lives until the kernel returns. `defk-4b2` bounded the +total with `max_owned_bytes`, so a copy or an alloc inside a loop raises +"out of memory" instead of taking the node down, but the buffers still +accumulate: `b = a; a[i] = x; ... b[0]` in a loop body copies `a` every +iteration (docs/language.md, Copies), and each copy is kept until return +although only the latest two are ever read again. Elixir would +garbage-collect the rest. + +Two facts of the compiler decide what can be released safely: + +- The copy analysis (`Defk.Copies`) already tracks which bindings *may* + share a buffer: alias classes for the arrays an `if`, `cond` or `case` + gives, and `{id, buf}` pairs for what is live after a statement and after + a block. A buffer no live pair may share is dead. +- The analysis gives a loop accumulator a fresh buffer ref whether or not + its initial value was copied, so one physical buffer can carry two refs + (the dead outer binding's and the accumulator's). That is harmless for + the copy decision (the outer binding is dead, or a copy was made) but + unsound for a release: a buffer could be released through one ref while + the other is live. + +Elixir mode enforces the same `max_owned_bytes` on the bytes `alloc/2` +asks for, and docs/language.md promises that native code's total is never +below Elixir mode's, so an `alloc/2` in a loop raises at the same iteration +in both modes. Releasing a dead `alloc/2` buffer natively would invert +that: native code would succeed where Elixir mode raises. + +## Decision + +1. **Only copies are released early.** A buffer is released where the last + binding of the block that holds it dies (after a statement nothing + later reads it, or after the block's last statement with the block's + value bound to locals first), when the buffer was made by a copy and no + live holder may be the same buffer. `alloc/2` buffers, bitstring + accumulators and aligned input copies live until return, as before, so + the budget agreement with Elixir mode stays as written. +2. **One physical buffer, one "may be the same" class.** `Defk.Copies` + records, beside the alias classes, which buffers are physically one + (`phys`: an accumulator and its source when no copy was made, the + accumulator and its body binding and its result, a merged value and + each arm's buffer) and which were made by a copy (`origin`). The release + test asks `may_same?/3` over both. An accumulator's buffer counts as a + copy only when its initial value was copied (or was a copy) and every + new value the body gives is a copy: the body is lowered under that + assumption and lowered again without it when a new value breaks it, + through the same backtracking as the accumulator facts. +3. **The runtime keeps the slot.** `defk_disown(ctx, data)` releases the + binary, subtracts its bytes from the budget and puts the slot on a free + list threaded through the empty slots, which `defk_own_raw` fills before + taking a new one, so a loop's table stays small. The array value stays + two words (data, length); the slot is found by a backward scan, since + the buffer that dies is usually a recent one. A slot index inside every + array value was rejected: it would touch every array operation. +4. **A stale pair does not hold a buffer.** A `{id, buf}` pair live after a + block whose binding was since copied in place (the same id with a + buffer that is not physically `buf`) names a buffer the binding no + longer holds. An accumulator binding stands in for the outer binding it + took its buffer from without a copy (`take_over/3`), so the loop + statement's reads of that name, resolved before the body bound it + again, do not keep the old buffer alive. + +Two calls the implementation made beyond the owner's "copies only": + +- The release point is every block whose bindings die with it (an arm, a + loop body, an inlined callee's body), as the proposal recommended, and + also the kernel's own body, which the proposal said needed nothing: + there a dead copy is released before the next statement instead of at + return (`copy_kept` in `test/owned_budget_test.exs` releases the hoisted + copy of its input after the loop). One rule, one code path. +- A plain nested block (`x = (a; b)`) releases nothing: its bindings live + on in the block around it. + +## Consequences + +- A per-iteration copy whose binding dies holds at most two buffers at a + time instead of one per iteration; `test/owned_budget_test.exs` runs + 1000 iterations of a 1 KiB copy under an 8 KiB budget, natively and in + Elixir mode, also under forced yields and as a property. +- The release is static and conservative: a buffer an `if`/`cond`/`case` + value may hold, or one a loop accumulator may hold when not every value + is a copy, is never released early (`copy_kept` in the same test still + raises past the budget). +- A released binding's C local is stale at the next spill; a spill is a + store, never a dereference, and the binding is bound again before any + read. The generated C of a kernel without a dying copy is unchanged but + for `free1 = 0` in the entry. +- The report says where each copy is released; a copy in a loop that is + not released is still noted as kept per iteration. diff --git a/docs/guides/runtime-contract.md b/docs/guides/runtime-contract.md index 2978d024..d27c658a 100644 --- a/docs/guides/runtime-contract.md +++ b/docs/guides/runtime-contract.md @@ -113,8 +113,10 @@ kernel is suspended, nothing leaks: `Mod.__defk_stats__().frames_live` goes back down. `alloc/2` raises `ArgumentError` for a length over 2^48 bytes or one the allocator cannot provide, instead of failing later. -A kernel's arrays live until it returns, so an `alloc` or a copy in a loop -body keeps one per iteration. On an operating system that overcommits +A kernel's arrays live until it returns, except a copy whose last variable +died, which is released then ([Copies](../language.md#copies)); so an +`alloc` in a loop body keeps one per iteration, and a copy whose variable +is read in the next iteration does too. On an operating system that overcommits memory, the node would be killed long before the allocator failed, so each call has a budget: an allocation that would take the bytes it holds past `use Defk, max_owned_bytes: n` (by default half of physical memory) raises diff --git a/docs/language.md b/docs/language.md index a2791886..cfdd49af 100644 --- a/docs/language.md +++ b/docs/language.md @@ -1601,20 +1601,36 @@ or clause head that chose it, which ran before it. It is conservative, so a kernel may copy where Elixir would not have to (for example when `b` is read earlier in the loop body than the write, which is not a read after it). The fast-path report (`Defk.Report`) notes each copy, and -`Mod.__defk_c_source__/0` shows them as `defk_copy_*` calls. Like `alloc`, -every copy lives until the kernel returns, so a copy in a loop body keeps -one array per iteration (the report notes that too), counted against the -[owned-buffer budget](#the-owned-buffer-budget). +`Mod.__defk_c_source__/0` shows them as `defk_copy_*` calls. + +A copy is released before the kernel returns when its last variable dies: +when the block that bound the variable (an `if`, `cond` or `case` arm, a +loop body, a called kernel's body, the kernel's own) reaches a statement +after which nothing reads it, and no other variable that may share the +buffer is read later, in the block or after it. So the copy that `b = a; a[i] += x; ... b[0]` makes every iteration no longer keeps one buffer per +iteration: when `b` dies at the end of the iteration, the previous +iteration's copy goes with it, and the loop holds two at a time. The test +is static and conservative, like the copy decision: a buffer an `if`, +`cond` or `case` value may hold (`c = if p, do: b, else: a`) is never +released early, and nor is one a loop accumulator may hold unless every +value of the accumulator is a copy. Only copies are released early: an +`alloc/2` buffer lives until the kernel returns, so that both modes raise +at the same allocation under the +[owned-buffer budget](#the-owned-buffer-budget). The report notes each +release, and `__defk_c_source__/0` shows them as `defk_disown` calls; a +copy in a loop that is not released is noted as kept per iteration. ### The owned-buffer budget Every array a call allocates (`alloc/2`), every copy above, the buffer of a [bitstring accumulator](#the-bitstring-accumulator) and the aligned copy of -a misaligned input are the kernel's own until it returns. Elixir would -garbage-collect an array no variable sees any more; a kernel frees them all -at once when it returns, raises, or its process dies. So a call keeps a -running total of the bytes it holds, and an allocation that would take it -past the module's budget raises `ArgumentError`: +a misaligned input are the kernel's own until it returns, except a copy +whose last variable died (released then, see [Copies](#copies)). Elixir +would garbage-collect an array no variable sees any more; a kernel frees +the rest all at once when it returns, raises, or its process dies. So a +call keeps a running total of the bytes it holds, and an allocation that +would take it past the module's budget raises `ArgumentError`: ``` ** (ArgumentError) out of memory (allocating 1024 bytes would take this call's buffers past max_owned_bytes, 8192 bytes) @@ -1637,7 +1653,8 @@ fails, which would break the [runtime contract](guides/runtime-contract.md). the host's memory). `Mod.__defk_stats__().max_owned_bytes` shows the budget. - The count is per call: concurrent calls each have their own, a call after a raise starts from zero, and a yield keeps it. An array a kernel - returns counts until it returns. + returns counts until it returns; a copy released early stops counting + when it is released. - Elixir mode enforces the same budget on `alloc/2`: it adds up the bytes every `alloc/2` of the call asks for (in called kernels too), against the same `max_owned_bytes` (it reads `__defk_stats__().max_owned_bytes`), and raises @@ -1647,7 +1664,9 @@ fails, which would break the [runtime contract](guides/runtime-contract.md). copies, bitstring accumulator buffers or aligned input copies (it garbage-collects those, as Elixir does), so a call whose copies pass the budget raises in native code and succeeds in Elixir mode: its total is - never above native code's, so never the reverse. + never above native code's, so never the reverse. That is why only copies + are released early: releasing an `alloc/2` buffer natively would put + native code's total below Elixir mode's. - A single allocation of 2^47 bytes or more, which no allocator serves, raises the plain "out of memory (allocating N bytes)" of [Allocating](#allocating) instead, the same in both modes. diff --git a/docs/performance.md b/docs/performance.md index 5dabdc24..9e328de5 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -3124,6 +3124,45 @@ the loop's exit path, once per block) beside the 16 MB time; `DEFK_TICKS` in `lib/defk/compiler.ex` (`sp_macro/3`, `run_preamble/1`). - **Status:** measured. +### 6.47 Releasing a dead copy early makes a per-iteration copy 2-7x cheaper: the slot and the binary are reused instead of growing the table + +Since `defk-c1e` a copy is released where its last variable dies +(docs/language.md, Copies; ADR-0012), so `b = a; a[i] = x; ... b[0]` in a +loop body holds two copies at a time instead of one per iteration. The +release itself is one call (a backward scan of the owned table, usually +one step, and `enif_release_binary`), and it pays for itself at once: the +next copy takes the freed slot and, from the allocator, usually the freed +binary, so the table never grows and the copy stays in cache. Before, each +iteration took a new binary and, every power of two, a bigger table. + +- **Numbers** (`copy_loop` of + [`test/owned_budget_test.exs`](https://github.com/DanielLuning/defk/blob/main/test/owned_budget_test.exs), median of 7 calls, ns per + iteration): 16 i64 elements, 1M iterations: `main` 92.3, with the + release 41.7 (2.2x); 1024 elements, 100k iterations: `main` 864, with + the release 130 (6.6x). The 6.25 figure (~50 ns per copy iteration at 32 + bytes) was measured at 200k iterations, before the table and the + allocator's free lists had grown far. +- **Conditions:** M-series Mac on battery, one session, no quiet check; + provisional. +- **Source:** `defk-c1e`, a scratch script of `copy_loop` on both + checkouts; `defk_disown` and `defk_own_raw` in [`priv/defk_runtime.h`](https://github.com/DanielLuning/defk/blob/main/priv/defk_runtime.h), + `Copies.dead/4` in [`lib/defk/copies.ex`](https://github.com/DanielLuning/defk/blob/main/lib/defk/copies.ex). +- **Status:** provisional. + + +```elixir +# released: b dies at the end of the iteration, and the previous +# iteration's copy of a with it +for i <- 0..(n - 1)//1, reduce: 0 do + acc -> + b = a + a[0] = i + acc + b[0] + a[0] +end +# kept until return: c may be b or a, so neither buffer is released +c = if i > 2, do: b, else: a +``` + --- ## 7. When a hand-written or wrapper NIF is still the answer diff --git a/lib/defk/compiler.ex b/lib/defk/compiler.ex index 5d2fd080..0d5a4d8f 100644 --- a/lib/defk/compiler.ex +++ b/lib/defk/compiler.ex @@ -375,7 +375,7 @@ defmodule Defk.Compiler do case Code.string_to_quoted(src) do # the statements are lowered in the block of the statement env is at {:ok, ast} -> - probe(fn -> stmt_list(exprs(ast), Copies.block_liveness(exprs(ast)), env) end) != + probe(fn -> stmt_list(exprs(ast), Copies.block_liveness(exprs(ast)), env, false) end) != :error _ -> @@ -908,22 +908,27 @@ defmodule Defk.Compiler do # A block whose last expression is its value. It is nested in the statement # `env` is at (see "array values"), or is a kernel's body. defp lower_block(ast, env) do - {s, v, _env} = lower_block_env(ast, env) + {s, v, _env} = lower_block_env(ast, env, true) {s, v} end - # lower_block/2 and the env the block ends in (nested, see leave/2) - defp lower_block_env(ast, env) do + # lower_block/2 and the env the block ends in (nested, see leave/2). + # `release?`: the block's bindings die with it (an arm, a loop body, a + # callee's body, the kernel's), so a copy of its own is released where its + # last binding dies (release_after/2, release_tail/2); a plain block's + # bindings live on in the block around it (leave/2), so there nothing is. + defp lower_block_env(ast, env, release?) do es = exprs(ast) {init, [last]} = Enum.split(es, -1) lives = Copies.block_liveness(es) - {s1, env} = stmt_list(init, lives, nested(env, ast)) + {s1, env} = stmt_list(init, lives, nested(env, ast), release?) {s2, v, env} = in_ctx(last, fn -> {sh, env} = enter(last, List.last(lives), env) {s, v, env} = last_value(last, env) - {[sh, s], v, env} + {sr, v, env} = if release?, do: release_tail(v, env), else: {[], v, env} + {[sh, s, sr], v, env} end) {[s1, s2], v, env} @@ -937,7 +942,7 @@ defmodule Defk.Compiler do # a nested block's value, and the env with its bindings; any other expression # binds nothing defp lower_expr_env({:__block__, _, _} = b, env) do - {s, v, inner} = lower_block_env(b, env) + {s, v, inner} = lower_block_env(b, env, false) {s, v, leave(inner, env)} end @@ -967,25 +972,139 @@ defmodule Defk.Compiler do # A block evaluated only for effect, nested like lower_block's. defp lower_stmts(ast, env) do es = exprs(ast) - stmt_list(es, Copies.block_liveness(es), nested(env, ast)) + stmt_list(es, Copies.block_liveness(es), nested(env, ast), true) end # statements `es` of one block; `lives` has, for each, the names read after - # it and the names it reads (Copies.block_liveness/1) - defp stmt_list(es, lives, env) do + # it and the names it reads (Copies.block_liveness/1); `release?` as + # lower_block_env/3's + defp stmt_list(es, lives, env, release?) do Enum.zip(AllocZeroing.mark(es), lives) |> Enum.reduce({[], env}, fn {e, live}, {acc, env} -> {s, env} = recover(e, env, fn -> {sh, env} = enter(e, live, env) {s, env} = lower_stmt(e, env) - {[sh, s], env} + {sr, env} = if release?, do: release_after(env), else: {[], env} + {[sh, s, sr], env} end) {[acc, s], env} end) end + # ------------------------------------------------------ early release --- + # A copy's buffer lives until the kernel returns, like every owned buffer, + # unless its last binding dies first: then it is released there + # (defk_disown), and a copy made every iteration of a loop holds one + # buffer at a time instead of one per iteration. Which bindings die, and + # whether a live holder may still see the buffer, is Copies.dead/4: the + # block's own bindings, not read after this statement, holding a copy no + # live {id, buf} pair and no buffer of the statement's value may be. Only + # copies: an alloc/2 buffer is counted by Elixir mode's budget as well, + # and releasing it natively would make the two modes raise at different + # allocations (docs/language.md, the owned-buffer budget). + # + # After a statement (release_after/2) nothing of the dying bindings is + # read again. After a block's last statement (release_tail/2) the block's + # value is still read by whatever consumes the block, so the value's + # scalar C is bound to locals first (settled/1); its arrays are pointer + # moves whose buffers are holders, never released here. + + defp release_after(env) do + case Copies.dead(copies(), env.scope, env.vars, []) do + [] -> {[], env} + dead -> release(dead, env) + end + end + + defp release_tail(v, env) do + case Copies.dead(copies(), env.scope, env.vars, bufs_of(v)) do + [] -> + {[], v, env} + + dead -> + {sv, v} = settled(v) + {sr, env} = release(dead, env) + {[sv, sr], v, env} + end + end + + # the buffers an array value, or the arrays of a tuple value, may hold + defp bufs_of(%{elems: elems}), do: Enum.flat_map(elems, &bufs_of/1) + defp bufs_of(%{t: {:arr, _}} = v), do: if(v[:buf], do: [v.buf], else: []) + defp bufs_of(_v), do: [] + + # `v` with each scalar whose C is an expression bound to a local of its + # own: {the assignments, the value} + defp settled(%{elems: elems} = v) do + {ss, elems} = Enum.map(elems, &settled/1) |> Enum.unzip() + {ss, %{v | elems: elems}} + end + + defp settled(%{t: {:arr, _}} = v), do: {[], v} + + defp settled(%{c: c, t: t} = v) do + if Regex.match?(~r/^[A-Za-z_]\w*$/, c) do + {[], v} + else + tmp = fresh("value", t) + srec_assign(tmp, c) + {"#{tmp} = #{c};\n", Map.drop(%{v | c: tmp, var: tmp}, [:pure_read])} + end + end + + # Release the buffers of the dying bindings `dead` ({name, binding}) and + # take the bindings out of the env. Each release is noted in the report. + # A copy noted as kept per iteration (note_per_iteration/2) in the same + # loop is re-noted: released when the dying binding holds that copy, or + # the previous iteration's when it holds what the copy was made from (`b + # = a; a[i] = v` copies `a` in place each iteration, and `b` still holds + # the copy of the iteration before), so the loop holds one or two of + # these copies at a time, not one per iteration. + defp release(dead, env) do + loop = List.first(low(:loops)) + + s = + for {name, b} <- dead do + noted = + Enum.find(low(:per_iter), fn {cbuf, {_at, l}} -> + l == loop and (cbuf == b.buf or Copies.carries?(copies(), cbuf, b.buf)) + end) + + case noted do + nil -> + note( + :owned, + env.scope.stmt, + :released, + "#{name}'s buffer, a copy, is released here: nothing reads it after this statement" + ) + + {cbuf, {at, _}} -> + drop_note(at, :per_iteration) + low_update(:per_iter, &Map.delete(&1, cbuf)) + + held = + if cbuf == b.buf, + do: "the loop holds one at a time", + else: + "the previous iteration's goes where its last binding dies, so the loop holds at most two at a time" + + note( + :owned, + at, + :released, + "this copy is inside a loop and is released before the kernel returns: #{held}" + ) + end + + "defk_disown(ctx, #{b.c}.data);\n" + end + + {s, %{env | vars: Map.drop(env.vars, Enum.map(dead, &elem(&1, 0)))}} + end + # run `fun` with `e` as the statement that literals inside it are located in defp in_ctx(e, fun) do if positioned?(e) do @@ -1366,7 +1485,8 @@ defmodule Defk.Compiler do defp enter(e, {live, reads}, env) do names = Copies.live_names(e, live) {sh, env} = hoist(e, names, reads, env) - {sh, %{env | scope: Copies.enter_stmt(env.scope, e, names, reads, env.vars)}} + scope = %{Copies.enter_stmt(env.scope, e, names, reads, env.vars) | read_after: live} + {sh, %{env | scope: scope}} end # A variable bound outside the branches and loop bodies of `e` and written @@ -1413,6 +1533,7 @@ defmodule Defk.Compiler do {:arr, e} = a.t buf = carried(make_ref(), a[:buf]) + copies_put(Copies.mark_copy(copies(), buf)) if a.owned and a[:var] != nil do {copy_stmts(a.c, a.c, e), Map.put(a, :buf, buf)} @@ -1526,6 +1647,10 @@ defmodule Defk.Compiler do {s, b} = copied(a, n) note(:copy, arr, :copied, "#{n} is copied before this write: #{copy_why(a)}") + + low(:loops) != [] && + low_update(:per_iter, &Map.put(&1, b.buf, {arr, List.first(low(:loops))})) + {s, b, %{env | vars: Map.put(env.vars, n, b), facts: Facts.bind(env.facts, b)}} _ -> @@ -4902,12 +5027,7 @@ defmodule Defk.Compiler do acc = Facts.len_facts_of( - %{ - t: v.t, - owned: true, - fin: v[:fin] || false, - buf: carried(make_ref(), v[:buf]) - }, + Map.merge(%{t: v.t, owned: true, fin: v[:fin] || false}, acc_buffer(ic, v)), v ) @@ -4924,6 +5044,30 @@ defmodule Defk.Compiler do ], Enum.map(accs, &elem(&1, 0))} end + # The buffer of an array accumulator whose initial value `v` is lowered + # to `ic` (loop_owned/2). Copied: the accumulator's buffer is a copy, and + # the body is lowered assuming its new values are too (lost_copy?/2 sends + # it back when one is not). Else it is physically the initial value's + # buffer (Copies.same/3), a copy when that was one, and `from` names the + # binding it came from, which the body's accumulator binding stands in + # for (Copies.take_over/3). + defp acc_buffer(ic, v) do + copied? = copy_parts(ic) != nil + buf = carried(make_ref(), v[:buf]) + + copies_put( + if copied?, + do: Copies.mark_copy(copies(), buf), + else: Copies.same(copies(), buf, v[:buf]) + ) + + %{ + buf: buf, + copy: copied? or Copies.copy_origin?(copies(), v[:buf]), + from: if(copied?, do: nil, else: v[:id]) + } + end + # an i64 literal >= 0 may become a u64 accumulator (u64_accs/2): marked # with the fold's ref defp u64_candidates(inits, items, ref) when length(items) == length(inits) do @@ -4960,9 +5104,12 @@ defmodule Defk.Compiler do end end - defp drop_facts({name, c, a} = acc, weak) do + defp drop_facts({name, c, a}, weak) do + # the copy assumption is dropped on its own: the length facts stay + a = if {c, :copy} in weak, do: %{a | copy: false}, else: a + cond do - c not in weak -> acc + c not in weak -> {name, c, a} Facts.weakest?(a) -> {name, c, Facts.weaken(%{a | fin: false})} true -> {name, c, Facts.weaken(a)} end @@ -4983,8 +5130,14 @@ defmodule Defk.Compiler do (e in [:f64, :f32] and a.fin and not r[:fin]) or not Facts.covers?(a, r) end + # the body assumed accumulator `a` holds a copy (reduce_accs/7) and its + # new value `r` is not one: noted apart from the facts (drop_facts/2) + defp lost_copy?(a, r), do: a[:copy] == true and not Copies.copy_origin?(copies(), r[:buf]) + # an array accumulator with facts a new value may not have - defp facts?(%{t: {:arr, e}} = a), do: (e in [:f64, :f32] and a.fin) or Facts.carries?(a) + defp facts?(%{t: {:arr, e}} = a), + do: (e in [:f64, :f32] and a.fin) or a[:copy] == true or Facts.carries?(a) + defp facts?(_a), do: false defp acc_env(accs, env) do @@ -4998,8 +5151,17 @@ defmodule Defk.Compiler do # the outer variable it shadows) match?({:arr, _}, a.t) -> buf = carried(make_ref(), a[:buf]) - b = Map.merge(a, %{c: c, id: make_ref(), buf: buf, depth: Copies.depth(e.scope) + 1}) - %{put_var(e, name, b) | scope: Copies.unguard(e.scope, name)} + id = make_ref() + b = Map.merge(a, %{c: c, id: id, buf: buf, depth: Copies.depth(e.scope) + 1}) + # physically the accumulator's buffer, a copy when the + # accumulator's values are (reduce_accs/7); the outer binding it + # took the buffer from (no copy) is read in the loop only through + # this binding, so the scope's pairs of that binding are its now + st = Copies.same(copies(), buf, a[:buf]) + copies_put(if a[:copy], do: Copies.mark_copy(st, buf), else: st) + scope = Copies.unguard(e.scope, name) + scope = if a[:from], do: Copies.take_over(scope, a.from, id), else: scope + %{put_var(e, name, b) | scope: scope} # an accumulator that may become a u64 is marked (u64_mix!/2) a[:u64_ref] -> @@ -5058,6 +5220,9 @@ defmodule Defk.Compiler do lost_facts?(a, r) && low_update(:acc_fix, &Map.update!(&1, fix, fn set -> MapSet.put(set, c) end)) + lost_copy?(a, r) && + low_update(:acc_fix, &Map.update!(&1, fix, fn set -> MapSet.put(set, {c, :copy}) end)) + a[:buf] && not Copies.carries?(copies(), r[:buf], a.buf) && copies_put(Copies.cut_carry(copies(), a.buf)) @@ -5099,8 +5264,12 @@ defmodule Defk.Compiler do defp acc_result(accs), do: %{t: {:tuple, Enum.map(accs, &elem(&1, 2).t)}, elems: Enum.map(accs, &acc_value/1)} - defp acc_value({_, c, %{t: {:arr, _}} = a}), - do: Map.merge(a, %{c: c, var: c, buf: carried(make_ref(), a[:buf])}) + defp acc_value({_, c, %{t: {:arr, _}} = a}) do + buf = carried(make_ref(), a[:buf]) + st = Copies.same(copies(), buf, a[:buf]) + copies_put(if a[:copy], do: Copies.mark_copy(st, buf), else: st) + Map.merge(a, %{c: c, var: c, buf: buf}) + end defp acc_value({_, c, a}), do: val(c, a.t, c) @@ -5112,7 +5281,7 @@ defmodule Defk.Compiler do es = exprs(ast) {init, [last]} = Enum.split(es, -1) lives = Copies.block_liveness(es) - {s, env} = stmt_list(init, lives, nested(env, ast)) + {s, env} = stmt_list(init, lives, nested(env, ast), true) [ s, @@ -6038,7 +6207,7 @@ defmodule Defk.Compiler do [ "static ERL_NIF_TERM nif_#{k.cid}(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]) {\n", "#{f} local;\nERL_NIF_TERM res;\n", - "local.ctx.err = 0;\nlocal.ctx.owned = NULL;\nlocal.ctx.nowned = local.ctx.cap = 0;\nlocal.ctx.owned_bytes = 0;\nlocal.ctx.finished = local.ctx.is_resource = 0;\n", + "local.ctx.err = 0;\nlocal.ctx.owned = NULL;\nlocal.ctx.nowned = local.ctx.cap = local.ctx.free1 = 0;\nlocal.ctx.owned_bytes = 0;\nlocal.ctx.finished = local.ctx.is_resource = 0;\n", "local.resume = 0;\nmemset(local.copy, 0, sizeof local.copy);\n", # the run-queue hint, when the NIF takes one, is the last argument "defk_slice_init(env, &local.ctx, argc > #{k.arity} ? &argv[#{k.arity}] : NULL);\n", @@ -6226,6 +6395,20 @@ defmodule Defk.Compiler do note_per_iteration(topic, at) end + # take the kernel's :owned note with `verdict` at `at` out of the report + defp drop_note(at, verdict) do + {kernel, _yield?} = low(:report_kernel) + line = note_line(at) + + ctx_update( + :report, + &Enum.reject(&1, fn + {^kernel, ^line, {:owned, ^verdict, _}} -> true + _ -> false + end) + ) + end + # An alloc/2 or a copy runs inside a loop: every owned buffer lives until # the kernel returns, so the loop keeps one per iteration, bounded only by # max_owned_bytes. A copy hoisted before a loop is lowered outside it diff --git a/lib/defk/compiler/ctx.ex b/lib/defk/compiler/ctx.ex index 530967ca..44aa68c4 100644 --- a/lib/defk/compiler/ctx.ex +++ b/lib/defk/compiler/ctx.ex @@ -46,6 +46,7 @@ defmodule Defk.Compiler.Ctx do report_loop: nil, acc_fix: %{}, copies: %Defk.Copies{}, + per_iter: %{}, strip: %Defk.Strip{}, operand_binds: [], guard: nil, diff --git a/lib/defk/copies.ex b/lib/defk/copies.ex index 8778d1fe..2d5be2e2 100644 --- a/lib/defk/copies.ex +++ b/lib/defk/copies.ex @@ -11,12 +11,14 @@ defmodule Defk.Copies do # ------------------------------------------------- the tracking state --- # What one kernel's lowering has recorded so far: the writes log (for # unreturned_writes!), buffer carrying (:from, a copy or accumulator holds - # what its source held) and alias classes (buffers an if/cond/case made - # indistinguishable). The module is pure: state in, state out; the - # compiler keeps the current state as one field of its lowering ctx, so - # its backtracking snapshot covers it. + # what its source held), alias classes (buffers an if/cond/case made + # indistinguishable), the buffers that are copies (:origin, for the early + # release) and the buffers that are physically one (:phys, two refs the + # lowering gave one buffer without a copy). The module is pure: state in, + # state out; the compiler keeps the current state as one field of its + # lowering ctx, so its backtracking snapshot covers it. - defstruct writes: [], from: %{}, alias: %{} + defstruct writes: [], from: %{}, alias: %{}, origin: %{}, phys: %{} def new, do: %__MODULE__{} @@ -48,7 +50,9 @@ defmodule Defk.Copies do new = make_ref() classes = for b <- bufs, b != nil, uniq: true, do: alias_class(st, b) m = Map.new(st.alias, fn {b, c} -> {b, if(c in classes, do: new, else: c)} end) - {new, %{st | alias: Enum.reduce(classes, m, &Map.put(&2, &1, new))}} + st = %{st | alias: Enum.reduce(classes, m, &Map.put(&2, &1, new))} + # the merged value may be any arm's buffer: physically one with each + {new, Enum.reduce(classes, st, &phys_union(&2, new, &1))} end # another live binding may see array a's buffer @@ -57,6 +61,83 @@ defmodule Defk.Copies do buf != nil and Enum.any?(live, fn {id, b} -> alias_class(st, b) == buf and id != a[:id] end) end + # ------------------------------------------------------ early release --- + # A copy's buffer is released before the kernel returns (defk_disown) + # where its last binding dies, so a copy made every iteration of a loop + # holds one buffer at a time. Two facts beyond the alias classes decide it: + # + # origin - the buffers a copy made (copied/2 in the compiler, an + # accumulator whose values are copies): only those are released; + # an alloc/2 buffer is counted by Elixir mode's budget too and + # stays until return, so the two modes keep raising at the same + # allocation (docs/language.md, the owned-buffer budget) + # phys - buffers that are physically one: a loop accumulator and the + # array it started from when no copy was made, the accumulator + # and its body binding and its result, a merged value and each + # arm's buffer. A union-find over the alias classes, so "may be + # one buffer" is one question, may_same?/3. + # + # dead/4 asks: which bindings of the current block die here, holding a + # copy no live holder may share? + + # buffer `buf` was made by a copy + def mark_copy(st, nil), do: st + def mark_copy(st, buf), do: %{st | origin: Map.put(st.origin, buf, true)} + + def copy_origin?(_st, nil), do: false + def copy_origin?(st, buf), do: Map.get(st.origin, buf, false) + + # buffers `a` and `b` are physically one (no copy between them) + def same(st, a, b) when a == nil or b == nil, do: st + def same(st, a, b), do: phys_union(st, alias_class(st, a), alias_class(st, b)) + + defp phys_union(st, a, b) do + {ra, rb} = {phys_root(st, a), phys_root(st, b)} + if ra == rb, do: st, else: %{st | phys: Map.put(st.phys, ra, rb)} + end + + defp phys_root(st, c) do + case Map.fetch(st.phys, c) do + {:ok, p} -> phys_root(st, p) + :error -> c + end + end + + # buffers `a` and `b` may be one buffer: one alias class, or physically one + def may_same?(_st, nil, _b), do: false + def may_same?(_st, _a, nil), do: false + + def may_same?(st, a, b), + do: phys_root(st, alias_class(st, a)) == phys_root(st, alias_class(st, b)) + + # The bindings in `vars` bound in the current block (its depth) that die + # after the current statement and hold a copy no live holder may share: + # {name, binding} pairs. A holder is a binding of a name read after the + # statement in its block (`read_after`, resolved in `vars` as they are + # after the statement, which may have bound the name), a {id, buf} pair + # live after the block (`out`), or a buffer of the statement's value + # (`escaping`, the block's value when the statement is its last). An + # `out` pair whose binding was copied in place since (the same id in + # `vars` with a buffer that is not physically this one) names a buffer + # that binding no longer holds, and does not count; a pair whose id is + # not in `vars` (an outer binding the block shadows) counts. + def dead(st, %{depth: depth, out: out, read_after: names}, vars, escaping) do + holders = + for({id, buf} <- out, not stale?(st, vars, id, buf), do: buf) ++ + for({_, buf} <- pairs(vars, names), do: buf) ++ + escaping + + for {name, %{depth: ^depth, owned: true, t: {:arr, _}, buf: buf} = b} <- vars, + name not in names, + copy_origin?(st, buf), + not Enum.any?(holders, &may_same?(st, buf, &1)), + do: {name, b} + end + + defp stale?(st, vars, id, buf) do + Enum.any?(vars, fn {_, v} -> v[:id] == id and not may_same?(st, v[:buf], buf) end) + end + # Buffer `new` holds what buffer `old` held (a copy of it, or a loop # accumulator started from it): a write to `old` is carried into `new`. # Recorded for unreturned_writes!/3; `new` is returned. @@ -254,7 +335,9 @@ defmodule Defk.Copies do # out - {id, buf} live after the current block # live - {id, buf} live after the current statement # nest - {id, buf} live for blocks nested in the current statement - # live_names - the names read after the current statement in its block + # live_names - the names read after the current statement in its block, + # less the names it binds (those name its new binding) + # read_after - the names read after the current statement in its block # stmt - the current statement # guard - array names a loop body must not bind again (read after it) # loop - the innermost loop construct @@ -264,6 +347,7 @@ defmodule Defk.Copies do live: MapSet.new(), nest: MapSet.new(), live_names: MapSet.new(), + read_after: MapSet.new(), stmt: nil, guard: MapSet.new(), loop: nil, @@ -459,6 +543,23 @@ defmodule Defk.Copies do %{s | live: s.nest, guard: guard, loop: node} end + # The accumulator binding `new` of a loop body stands in for the outer + # binding `old` it took the buffer from without a copy (reduce_accs): the + # pairs of `old` in the scope (the loop statement's reads of that name, + # resolved before the body bound it again) are `new`'s now, so a copy of + # `new` in place makes them stale (dead/4) instead of keeping the old + # buffer alive for a binding that is never read. + def take_over(%Scope{} = s, old, new) do + swap = fn pairs -> + MapSet.new(pairs, fn + {^old, buf} -> {new, buf} + p -> p + end) + end + + %{s | out: swap.(s.out), live: swap.(s.live), nest: swap.(s.nest)} + end + # {id, buf} live around `node`, a part of the current statement: what is # live after the statement, what the statement reads outside `node`, and # the names `more` (what a loop's generators read while it runs). diff --git a/priv/defk_runtime.h b/priv/defk_runtime.h index 10fc5aa8..3503a71f 100644 --- a/priv/defk_runtime.h +++ b/priv/defk_runtime.h @@ -27,6 +27,7 @@ * yields: DEFK_BUDGET0 DEFK_ZERO_TICKS defk_clock defk_clock_init defk_tick * defk_lost_resume defk_stat_yields * arguments and results: defk_get__arg defk_make_arr defk_make_arr_view + * early release of a dead copy: defk_disown * defk_make_bits defk_make_bool defk_make_f64 * arrays: defk_arr_ defk_alloc_ defk_alloc_raw_ defk_copy_ * defk_copy_raw_ defk_get_ defk_put_ defk_to_ defk_to_i64_elem @@ -162,6 +163,10 @@ typedef struct { size_t ulen; ErlNifBinary *owned; /* binaries we allocated (outputs, aligned copies) */ int nowned, cap; + /* the empty slots of `owned` (released early by defk_disown, or handed + over), as a list threaded through their `size` fields: slot + 1, 0 at + the end; defk_own_raw fills them before it takes a new slot */ + int free1; uint64_t owned_bytes; /* bytes of the live binaries in owned, at most defk_max_owned */ int finished; int is_resource; @@ -394,6 +399,21 @@ static void defk_poison_slack(const ErlNifBinary *b) { /* Owned binary: off-heap, stable address across yields and GC; uninitialised. */ static int defk_own_raw(defk_ctx *ctx, size_t bytes) { defk_owned_check(ctx, bytes); + if (ctx->free1) { + /* an empty slot: a buffer released early (defk_disown) or handed over */ + int slot = ctx->free1 - 1; + ErlNifBinary *b = &ctx->owned[slot]; + size_t next1 = b->size; + if (!enif_alloc_binary(bytes, b)) { + b->data = NULL; + b->size = next1; + defk_fail(ctx, DEFK_E_ARG, DEFK_MSG_OUT_OF_MEMORY_BYTES, bytes); + } + ctx->free1 = (int)next1; + DEFK_POISON_SLACK(b); + ctx->owned_bytes += bytes; + return slot; + } if (ctx->nowned == ctx->cap) { if (!ctx->owned) { /* the first buffers go in the frame's own slots */ @@ -425,6 +445,34 @@ static int defk_own_raw(defk_ctx *ctx, size_t bytes) { return ctx->nowned++; } +/* slot `i` of `owned` is empty: put it on the free list (see defk_ctx.free1) */ +static inline void defk_slot_free(defk_ctx *ctx, int i) { + ctx->owned[i].data = NULL; + ctx->owned[i].size = (size_t)ctx->free1; + ctx->free1 = i + 1; +} + +/* + * Release the owned buffer at `data` before the kernel returns: the compiler + * emits it where a copy's last binding dies (docs/language.md, Copies), so a + * copy made every iteration of a loop no longer accumulates until return. + * The slot is reused by the next buffer, so a loop's table stays small. The + * scan runs from the newest slot: the buffer that dies is usually a recent + * one. A `data` no slot holds (a zero-length copy, say) is left alone. + */ +static void defk_disown(defk_ctx *ctx, void *data) { + if (!data) return; + for (int i = ctx->nowned - 1; i >= 0; i--) { + ErlNifBinary *b = &ctx->owned[i]; + if (b->data == data) { + ctx->owned_bytes -= b->size; + enif_release_binary(b); + defk_slot_free(ctx, i); + return; + } + } +} + /* Owned binary of zeros. */ static int defk_own(defk_ctx *ctx, size_t bytes) { int slot = defk_own_raw(ctx, bytes); @@ -710,11 +758,13 @@ DEFK_RANGE_COPY_FN(f32, float) /* Hand an owned array back to the BEAM as a binary (ownership transfers). */ static ERL_NIF_TERM defk_make_arr(ErlNifEnv *env, defk_ctx *ctx, void *data) { - for (int i = 0; i < ctx->nowned; i++) { + /* data is NULL only for a zero-length array, which no slot holds: the + empty slots (data NULL) must not match it */ + for (int i = 0; data && i < ctx->nowned; i++) { if (ctx->owned[i].data == data) { ctx->owned_bytes -= ctx->owned[i].size; ERL_NIF_TERM t = enif_make_binary(env, &ctx->owned[i]); - ctx->owned[i].data = NULL; /* no longer ours to release */ + defk_slot_free(ctx, i); /* no longer ours to release */ return t; } } diff --git a/test/copies_test.exs b/test/copies_test.exs index 6e95fbca..46cb5dc1 100644 --- a/test/copies_test.exs +++ b/test/copies_test.exs @@ -436,4 +436,100 @@ defmodule Defk.CopiesTest do assert match == MapSet.new([:y]) end end + + # The early release (docs/language.md, Copies; ADR-0012): which bindings + # of a block die after a statement holding a copy no live holder may + # share. The scope is built by hand: `out` the {id, buf} pairs live after + # the block, `read_after` the names read after the statement. + describe "early release (dead/4, may_same?/3)" do + defp arr(buf, depth \\ 1), + do: %{id: make_ref(), buf: buf, t: {:arr, :i64}, owned: true, depth: depth} + + defp scope(out, read_after, depth \\ 1), + do: %Copies.Scope{depth: depth, out: MapSet.new(out), read_after: MapSet.new(read_after)} + + test "a copy whose binding is not read again, and no holder shares, dies" do + buf = make_ref() + st = Copies.mark_copy(Copies.new(), buf) + vars = %{b: arr(buf)} + + assert [{:b, _}] = Copies.dead(st, scope([], []), vars, []) + end + + test "an alloc's buffer never dies early: only copies are released" do + buf = make_ref() + vars = %{b: arr(buf)} + assert Copies.dead(Copies.new(), scope([], []), vars, []) == [] + end + + test "a name read after the statement holds its buffer, resolved after the statement" do + buf = make_ref() + st = Copies.mark_copy(Copies.new(), buf) + # `out2 = c`: out2 is bound by the statement and read after it + vars = %{c: arr(buf), out2: arr(buf)} + + assert Copies.dead(st, scope([], [:out2]), vars, []) == [] + end + + test "a pair live after the block holds its buffer, unless its binding was copied in place" do + buf = make_ref() + st = Copies.mark_copy(Copies.new(), buf) + a = arr(buf) + vars = %{b: arr(buf), a: a} + + # the outer a (bound at depth 0) still holds buf + assert Copies.dead(st, scope([{a.id, buf}], [], 1), %{vars | a: %{a | depth: 0}}, []) == [] + + # a was copied in place since: the same id, a new buffer + new = make_ref() + st = Copies.mark_copy(st, new) + vars = %{vars | a: %{a | buf: new, depth: 0}} + assert [{:b, _}] = Copies.dead(st, scope([{a.id, buf}], [], 1), vars, []) + end + + test "a pair whose id the block shadows (not in vars) still holds" do + buf = make_ref() + st = Copies.mark_copy(Copies.new(), buf) + vars = %{b: arr(buf)} + assert Copies.dead(st, scope([{make_ref(), buf}], []), vars, []) == [] + end + + test "the statement's value holds its buffers" do + buf = make_ref() + st = Copies.mark_copy(Copies.new(), buf) + vars = %{b: arr(buf)} + assert Copies.dead(st, scope([], []), vars, [buf]) == [] + end + + test "physically one buffer (same/3) and a merged value (merge_bufs/2) are holders" do + buf = make_ref() + acc = make_ref() + st = Copies.new() |> Copies.mark_copy(buf) |> Copies.same(acc, buf) + assert Copies.may_same?(st, acc, buf) + assert Copies.dead(st, scope([{make_ref(), acc}], []), %{b: arr(buf)}, []) == [] + + {merged, st} = Copies.merge_bufs(st, [buf, make_ref()]) + assert Copies.may_same?(st, merged, buf) + assert Copies.may_same?(st, merged, acc) + assert Copies.dead(st, scope([], []), %{b: arr(buf)}, [merged]) == [] + refute Copies.may_same?(st, make_ref(), buf) + refute Copies.may_same?(st, nil, buf) + end + + test "only the block's own bindings die, not an outer one it can see" do + buf = make_ref() + st = Copies.mark_copy(Copies.new(), buf) + assert Copies.dead(st, scope([], [], 2), %{b: arr(buf, 1)}, []) == [] + assert [{:b, _}] = Copies.dead(st, scope([], [], 2), %{b: arr(buf, 2)}, []) + end + + test "take_over/3 gives the shadowed outer binding's pairs to the accumulator's" do + {old, new, buf} = {make_ref(), make_ref(), make_ref()} + other = {make_ref(), make_ref()} + s = %Copies.Scope{out: MapSet.new([{old, buf}, other]), live: MapSet.new([{old, buf}])} + s = Copies.take_over(s, old, new) + assert s.out == MapSet.new([{new, buf}, other]) + assert s.live == MapSet.new([{new, buf}]) + end + end end diff --git a/test/owned_budget_test.exs b/test/owned_budget_test.exs index faee5243..9759a982 100644 --- a/test/owned_budget_test.exs +++ b/test/owned_budget_test.exs @@ -24,7 +24,9 @@ defmodule Defk.OwnedBudgetTest.K do end # `b = a` in the body, `a` written and `b` read after it: a copy of `a` - # every iteration (docs/language.md, Copies) + # every iteration (docs/language.md, Copies). `b` dies at the end of the + # iteration, so the previous iteration's copy is released there: the loop + # holds two copies at a time, not one per iteration defk copy_loop(a :: [i64], n :: i64) :: i64 do for i <- 0..(n - 1)//1, reduce: 0 do acc -> @@ -34,6 +36,58 @@ defmodule Defk.OwnedBudgetTest.K do end end + # the same shape with `a` as a reduce accumulator + defk reduce_copy(a :: [i64], n :: i64) :: {[i64], i64} do + for i <- 0..(n - 1)//1, reduce: {a, 0} do + {a, s} -> + b = a + a[0] = i + {a, s + b[0] + a[0]} + end + end + + # the copy and the dying binding inside an if arm + defk arm_copy(a :: [i64], n :: i64) :: i64 do + for i <- 0..(n - 1)//1, reduce: 0 do + acc -> + if i > 0 do + b = a + a[0] = i + acc + b[0] + else + acc + a[0] + end + end + end + + defk bump(a :: [i64]) :: [i64] do + a[0] = a[0] + 1 + a + end + + # a called kernel copies the caller's array (the caller reads it after); + # the copy comes back as `b` and dies with the iteration + defk callee_copy(a :: [i64], n :: i64) :: i64 do + for _ <- 0..(n - 1)//1, reduce: 0 do + acc -> + b = bump(a) + acc + b[0] + a[0] + end + end + + # the copies stay reachable: `c` is `b` or `a`, which the compiler cannot + # tell apart, so neither buffer is released and the loop keeps one copy + # per iteration until the kernel returns + defk copy_kept(a :: [i64], n :: i64) :: i64 do + for i <- 0..(n - 1)//1, reduce: 0 do + acc -> + b = a + a[0] = i + c = if i > 2, do: b, else: a + acc + c[0] + end + end + # one alloc outside any loop, returned: under the budget, then handed over defk fill(n :: i64) :: [i64] do out = alloc(i64, n) @@ -94,6 +148,7 @@ end defmodule Defk.OwnedBudgetTest do use ExUnit.Case, async: false + use ExUnitProperties import Defk.Test.Frames import Defk.TestKit @@ -149,15 +204,103 @@ defmodule Defk.OwnedBudgetTest do assert same_raise(K, :alloc_twice, [129]) =~ ~r/^out of memory \(allocating 1032 bytes/ end - test "a copy in a loop raises out of memory past the budget" do + test "a copy in a loop that stays reachable raises out of memory past the budget" do a = i64s(Enum.to_list(1..128)) - assert K.copy_loop(a, 3) == elixir(:copy_loop, [a, 3]) - assert_raise ArgumentError, @oom, fn -> K.copy_loop(a, 1000) end - # Elixir mode counts alloc/2 only: it garbage-collects the old copies - # a write in a `for` body is kept for the next iteration: b[0] is the - # previous i (1, the input's, at i = 0), a[0] is i - assert elixir(:copy_loop, [a, 1000]) == 1 + Enum.sum(for i <- 1..999, do: 2 * i - 1) - assert K.copy_loop(a, 7) == elixir(:copy_loop, [a, 7]) + assert K.copy_kept(a, 3) == elixir(:copy_kept, [a, 3]) + assert_raise ArgumentError, @oom, fn -> K.copy_kept(a, 1000) end + # Elixir mode counts alloc/2 only: it garbage-collects the old copies. + # c[0] is a[0] = i for i <= 2, then b[0], the previous i + assert elixir(:copy_kept, [a, 1000]) == 0 + 1 + 2 + Enum.sum(for i <- 3..999, do: i - 1) + assert K.copy_kept(a, 7) == elixir(:copy_kept, [a, 7]) + end + + describe "early release" do + # the dying binding's buffer is released before the kernel returns, so + # 1000 iterations of a 1 KiB copy fit an 8 KiB budget + test "a per-iteration copy whose binding dies is released: the loop no longer accumulates" do + a = i64s(Enum.to_list(1..128)) + # a write in a `for` body is kept for the next iteration: b[0] is the + # previous i (1, the input's, at i = 0), a[0] is i + expected = 1 + Enum.sum(for i <- 1..999, do: 2 * i - 1) + assert K.copy_loop(a, 1000) == expected + assert elixir(:copy_loop, [a, 1000]) == expected + assert K.reduce_copy(a, 1000) == elixir(:reduce_copy, [a, 1000]) + assert K.arm_copy(a, 1000) == elixir(:arm_copy, [a, 1000]) + assert K.callee_copy(a, 1000) == elixir(:callee_copy, [a, 1000]) + end + + defp kernel_c(name) do + K.__defk_c_source__() + |> String.split("/* ---- #{name}/2") + |> Enum.at(1) + |> String.split("/* ---- ") + |> hd() + end + + test "the release is one runtime call per dying binding, in the generated C" do + for name <- ["copy_loop", "reduce_copy", "arm_copy", "callee_copy"] do + assert kernel_c(name) =~ "defk_disown(ctx, ", name + end + + # copy_kept releases one buffer: the hoisted copy of the input, after + # the loop; nothing inside the body + [before_end, after_end] = String.split(kernel_c("copy_kept"), "/* defk:end */") + refute before_end =~ "defk_disown" + assert after_end =~ "defk_disown" + refute kernel_c("alloc_loop") =~ "defk_disown" + end + + test "under forced yields the released buffers and the budget agree with Elixir mode" do + a = i64s(Enum.to_list(1..128)) + + forced(K, fn -> + {res, count} = yields(K, fn -> K.copy_loop(a, 200) end) + assert res == elixir(:copy_loop, [a, 200]) + assert count > 0 + assert K.reduce_copy(a, 200) == elixir(:reduce_copy, [a, 200]) + assert K.arm_copy(a, 200) == elixir(:arm_copy, [a, 200]) + assert K.callee_copy(a, 200) == elixir(:callee_copy, [a, 200]) + assert_raise ArgumentError, @oom, fn -> K.copy_kept(a, 1000) end + end) + + assert_frames_live(K) + end + + property "every loop length and array agrees with Elixir mode under the budget" do + check all( + len <- StreamData.integer(1..64), + n <- StreamData.integer(0..300), + xs <- StreamData.list_of(StreamData.integer(-1000..1000), length: len), + max_runs: 40 + ) do + a = i64s(xs) + + for f <- [:copy_loop, :reduce_copy, :arm_copy, :callee_copy] do + assert apply(K, f, [a, n]) == elixir(f, [a, n]), "#{f}(#{len}, #{n})" + end + end + end + + test "notes the release in the fast-path report" do + released = fn kernel -> + for {^kernel, line, {:owned, :released, text}} <- K.__defk_report__(), do: {line, text} + end + + # the copy made each iteration, and the hoisted copy of the input, + # released after the loop + assert [{_, text}, {_, after_loop}] = Enum.sort(released.({:copy_loop, 2}), :desc) + assert text =~ "this copy is inside a loop and is released before the kernel returns" + assert text =~ "at most two at a time" + assert after_loop =~ "a's buffer, a copy, is released here" + assert owned_notes({:copy_loop, 2}) == [] + + assert [{_, text}] = released.({:callee_copy, 2}) + assert text =~ "released" + + assert [{_, text}] = released.({:copy_kept, 2}) + assert text =~ "a's buffer, a copy, is released here" + assert [{_, _}] = owned_notes({:copy_kept, 2}) + end end test "a returned array is counted while the kernel runs, and a call under the budget agrees" do @@ -198,8 +341,8 @@ defmodule Defk.OwnedBudgetTest do assert_raise ArgumentError, @oom, fn -> K.alloc_loop(9, 128) end assert K.alloc_loop(8, 128) == Enum.sum(0..7) a = i64s(Enum.to_list(1..128)) - assert K.copy_loop(a, 3) == elixir(:copy_loop, [a, 3]) - assert_raise ArgumentError, @oom, fn -> K.copy_loop(a, 1000) end + assert K.copy_kept(a, 3) == elixir(:copy_kept, [a, 3]) + assert_raise ArgumentError, @oom, fn -> K.copy_kept(a, 1000) end assert K.bits(1000) == elixir(:bits, [1000]) assert_raise ArgumentError, ~r/^out of memory/, fn -> K.bits(10_000) end end) @@ -256,7 +399,7 @@ defmodule Defk.OwnedBudgetTest do assert text =~ "Allocate before the loop and clear each pass" - assert [{_, text}] = owned_notes({:copy_loop, 2}) + assert [{_, text}] = owned_notes({:copy_kept, 2}) assert text =~ "this copy is inside a loop: each iteration keeps a buffer" assert Defk.Report.format(K) =~ "each iteration keeps a buffer until the kernel returns" end