From fd31b0389109881d0f86d0c118ca8b4bb3f5ab39 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20L=C3=BCning?= Date: Tue, 20 Oct 2026 15:28:37 +0200 Subject: [PATCH 1/3] defk-9p6.30: tape_csv, CSV to rows through a kernel tape; poor, the wrapper row stays closed Two tape forms against NimbleCSV.RFC4180 and rusty_csv 0.4.6: an [i64] tape of field events walked with binary_part, and a cleaned copy (byte 30 and 31 separators) split twice by :binary.split. The copy is faster, and still 0.81-0.93x NimbleCSV on a 20 MB export: the scan takes 28-36 ms, building the rows in Elixir 135-314 ms. rusty_csv builds the same sub-binaries in C, 3.5-4x faster than NimbleCSV. Variant timings collect the heap before each run, as Defk.Bench.report/1 does: without that, term building looked 1.5-4x cheaper. Pre-registered marginal, measured poor. --- docs/performance.md | 27 +++ examples/README.md | 9 +- examples/_tables.exs | 10 + examples/tape_csv.exs | 449 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 494 insertions(+), 1 deletion(-) create mode 100644 examples/tape_csv.exs diff --git a/docs/performance.md b/docs/performance.md index c69d565b..94efd818 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -2906,6 +2906,33 @@ output array would). - **Source:** [`bench/port_roundtrip.exs`](https://github.com/DanielLuning/defk/blob/main/bench/port_roundtrip.exs), [`bench/port_echo.c`](https://github.com/DanielLuning/defk/blob/main/bench/port_echo.c), `defk-7dm`. - **Status:** one quiet run; rerun with `defk-sgw.4`'s snapshot if the verdict cites it. +### 7.11 CSV to terms: a kernel tape plus Elixir row building loses to NimbleCSV; rusty_csv builds the sub-binaries in C, 3.5-4x faster + +A kernel can scan the CSV and hand Elixir flat events, but Elixir then +builds every row and field, and that half costs more than NimbleCSV's +whole parse. A NIF that makes the sub-binaries itself is the fast route. + +- **Numbers:** 20 MB, a 200k-row export with quoted fields: NimbleCSV + 201-238 ms; rusty_csv 0.4.6 57.6-57.7 ms; the kernel's cleaned copy + (separators as bytes 30 and 31, split twice by `:binary.split/3`) + 249-257 ms end to end, 0.81-0.93x. Its parts: the copy 36 ms, the two + splits 135 ms. An `[i64]` tape of start and length per field: scan 28 + ms, rows built from it with binary matching and `binary_part/3` 314 ms. + Walking 1.4M tape entries without building anything took 39 ms. At 100 + MB the copy is 2.4-2.65x NimbleCSV, whose garbage costs most there. +- **Heap state matters for term building:** without a GC between timed + runs, each run reused the heap the last one grew, and the rows from the + tape took 83-92 ms instead of 314 ms (NimbleCSV 134-138 instead of + 201-238). `Defk.Bench.report/1` collects before each run; variant loops + in a script must do the same. +- **What it means:** the 13x gap between scanning and term creation that + `string-workloads.md` measured holds for Elixir, not for C: rusty_csv's + sub-binaries cost little. For CSV or JSON to terms, use NimbleCSV or + Jason, or the NIF when the speed matters. +- **Conditions:** M-series Mac, OTP 29, Elixir 1.20.2, two runs. +- **Source:** [`tape_csv.exs`](https://github.com/DanielLuning/defk/blob/main/examples/tape_csv.exs), `defk-9p6.30`. +- **Status:** provisional. + --- ## 8. Myths we disproved and open questions diff --git a/examples/README.md b/examples/README.md index ad473e7b..52303795 100644 --- a/examples/README.md +++ b/examples/README.md @@ -193,6 +193,13 @@ has them in full. 5.7–5.9× faster than NimbleCSV with `String.to_float`; the kernel is 2.7× that on CSV, and about 4× on JSON against Jason, OTP's `:json` and the same parser. +- [`tape_csv.exs`](tape_csv.exs): CSV to NimbleCSV's rows of binaries + through a kernel tape (field events, or a cleaned copy split by + `:binary.split`). The scan takes 28–36 ms of 20 MB; building the rows in + Elixir costs more than NimbleCSV's whole parse, so the tape is poor + (0.81–0.93×). rusty_csv's NIF builds the same sub-binaries in C, 3.5–4× + faster than NimbleCSV. Benchmarks of term building must start each run + from a collected heap: without it the rows looked 1.5–4× cheaper. - [`lz4_snappy_decode.exs`](lz4_snappy_decode.exs): Snappy and LZ4 block decode from scratch on untrusted input, against snappyer, lz4_erl and nimble_lz4. Against the NIFs as installed from Hex (built with no `-O`) @@ -298,7 +305,7 @@ Nothing here is excluded on principle: each row names the condition that would r | Job | Wrapped today | Why not from scratch | What would change it | |---|---|---|---| | Signing, key exchange, secret-dependent crypto (secp256k1, ed25519, X25519, AEAD) | libsodium, secp256k1, ring via Rustler; `:crypto` | The one rule says nothing about timing: clang `-O3` may turn a select into a branch or a table lookup into a secret-dependent access, and a process dump or yielded frame holds key material. Public-input verification is allowed but gains little. | A constant-time mode: a restricted subset checked at compile time, codegen verified with dudect or ctgrind in CI, frame zeroisation, and an external audit. Until then, `:crypto`. | -| Parsers that produce trees or term streams (HTML, Markdown, tree-sitter, JSON/CSV to terms, protobuf, msgpack) | html5ever, comrak, tree-sitter, jsont, rusty_csv, protox | The product is BEAM terms and building them is the cost: `docs/research/string-workloads.md` measured a 13x gap between scanning and term creation, and a NIF builds terms serially at the same price. A kernel cannot build terms, and if it could it would be no faster. | A "tape" output (flat `[i64]` of kind/start/len events, of which a fused CSV parser is one form), plus a benchmark showing tape plus Elixir tree building beats Jason or NimbleCSV end to end. BPE tokenisation to `[i64]` ids is the exception: its product is packed, and so is a numeric CSV or JSON number array loaded into f64s (`numeric_loader`, good: about 2.7× and 4×). | +| Parsers that produce trees or term streams (HTML, Markdown, tree-sitter, JSON/CSV to terms, protobuf, msgpack) | html5ever, comrak, tree-sitter, jsont, rusty_csv, protox | The product is BEAM terms and building them is the cost: `docs/research/string-workloads.md` measured a 13x gap between scanning and term creation. [`tape_csv`](tape_csv.exs) measured the tape route: a kernel scans 20 MB of CSV in 28–36 ms, and Elixir then takes 135–314 ms to build the rows from its events, so the whole is 0.81–0.93× NimbleCSV (poor). rusty_csv builds the same sub-binaries in C, 3.5–4× faster than NimbleCSV: here the NIF builds terms far cheaper than Elixir does. A kernel cannot build terms. | Closed with evidence for a tape (`tape_csv`). It would take a kernel that builds the sub-binaries and lists itself, which the kernel subset leaves out. BPE tokenisation to `[i64]` ids is the exception: its product is packed, and so is a numeric CSV or JSON number array loaded into f64s (`numeric_loader`, good: about 2.7× and 4×). | | Regular expressions | `:re` (PCRE in C), Rust regex via Rustler | `:re` is already native and compiled once per pattern; a kernel cannot take a pattern as data without becoming an interpreter, and fixed-string search is `:binary.match`. | A specific pattern class with a known kernel form (glob, or a simple byte DFA compiled at Elixir compile time) and a benchmark showing `:re` loses on it. | | Large codecs and encoders (brotli, zstd encode, audio and video, PDF/typst rendering) | brotli, zstd, membrane_* FFmpeg/opus bindings, typst | The cost is spec surface and tuning, not the loop: 10^4-10^5 lines with SIMD everywhere. A kernel would be a worse implementation of a solved problem. | A narrow, named need: one compression level with fixed tables (zstd level 1 is the lz4-encode ticket plus FSE) or a decoder-only requirement for untrusted input. Each reopens one row, not the category. | | Systems: databases, dataframes, tensor compilers (SQLite, DuckDB, Polars/Explorer, XLA/EXLA, LibTorch) | exqlite, duckdbex, explorer, exla, torchx | The value is the library: query planners, columnar engines, autodiff, device backends. defk is a kernel, not a system. | Interop, not rewriting: zero-copy views over their packed buffers so a per-element loop beside them runs as a kernel. `docs/research/nx-boundary.md` says where that line is; file an interop ticket when a workload asks. | diff --git a/examples/_tables.exs b/examples/_tables.exs index 11274a3a..95085e81 100644 --- a/examples/_tables.exs +++ b/examples/_tables.exs @@ -160,6 +160,11 @@ job: "sort 1M packed keys", line: "{speedup:sort_to_list} when the caller needs a list back" }, + "tape_csv" => %{ + job: "CSV of 20 MB to NimbleCSV's rows, through a kernel tape", + native: "{native} rusty_csv", + line: "the scan is fast; building the rows in Elixir costs more than NimbleCSV's whole parse" + }, "utf8_valid" => %{ job: "validate 16 MB of UTF-8", line: "`String.valid?` is C already" @@ -247,6 +252,11 @@ "anything `:crypto` does (a SHA-256 chain) and memory-hard password hashing (Argon2: the C runs its lanes on threads)", examples: ["hash_chain", "argon2"] }, + %{ + text: + "CSV to rows of binaries through a kernel tape: building the terms is the cost (rusty_csv's NIF builds them in C, 3.5× NimbleCSV)", + examples: ["tape_csv"] + }, %{text: "one kernel call per element", examples: ["poor_tiny_calls"]}, %{ text: "data that arrives as lists: packing costs more than the whole `Enum.sum`", diff --git a/examples/tape_csv.exs b/examples/tape_csv.exs new file mode 100644 index 00000000..aeb805db --- /dev/null +++ b/examples/tape_csv.exs @@ -0,0 +1,449 @@ +# tape_csv: CSV to rows of binaries through a tape: a kernel scans and +# emits flat field events, Elixir builds the rows +# +# The question (examples/README.md, "Where a wrapper is the right choice"): +# parsers whose product is terms stay out of the catalog because building +# the terms is the cost. What would reopen the row: a tape output (flat +# events of start and length) plus Elixir tree building that beats +# NimbleCSV end to end. This example measures that, for CSV to the lists +# of binaries NimbleCSV.RFC4180.parse_string/2 returns. The numeric form, +# which builds no terms, is numeric_loader.exs. +# +# elixir examples/tape_csv.exs +# +# Pre-registered: marginal (the research: Elixir still builds every +# sub-binary). Measured: poor, and the expected verdict below says so. +# +# Two tape forms, both from the same RFC 4180 grammar as NimbleCSV +# (quoted fields with commas, doubled quotes and line breaks; \n or \r\n): +# [i64] tape scan/2: two words per field (start; length plus a "row +# ends" and a "has doubled quotes" flag), sized by a vectorised +# count of ',' and '\n'. Elixir walks it with binary matching +# and takes binary_part of the input per field. +# copy normalize/1: a cleaned copy of the input, separating commas +# as byte 31, row ends as byte 30, quotes removed and doubled +# quotes written once. Elixir splits it twice with +# :binary.split/3 (C), rows then fields. Bytes 30 and 31 in +# the input raise ArgumentError (this form's limit). +# The copy is the faster form and the verdict's defk side. +# +# Elixir baseline: first pass, NimbleCSV.RFC4180 (which already splits +# unquoted runs with :binary.split). Not tried: a hand-written Elixir +# parser. With defk behind NimbleCSV, a faster Elixir side only widens the +# gap. Native: rusty_csv 0.4.6 (Rust NIF, precompiled; SIMD scan, then +# sub-binary terms built in C on a dirty scheduler). +# +# Agreement: both tape forms, their Elixir mode (`elixir_mode: true`, +# Tape.Kernels.__defk_elixir__/2) and rusty_csv return exactly NimbleCSV's +# rows on 49 inputs (empty input, empty fields and lines, \r\n, quoted +# commas, doubled quotes and line breaks, 1 to 3000 rows); 6 bad inputs (a +# quote inside an unquoted field, an unclosed quote, a byte after a closing +# quote) raise on every side. +# +# Realistic size: 20 MB, a 200k-row customer export (id, name, email, city, +# amount, date, a note quoted in a third of the rows), with a sweep of 2k, +# 20k and 1M rows (100 MB). +# +# Timing: every run starts from a collected heap (Defk.Bench.report/1 does +# this, and so do the variant lines). Without the GC between runs, each run +# reuses the heap the previous one grew, and building the terms looked +# 1.5-4x cheaper than a caller sees it (the [i64] tape's rows 83-92 ms +# instead of 314 ms). +# +# Measured (M-series Mac, OTP 29, Elixir 1.20, 2026-10-20, best of 15; two +# runs, provisional until the stored run): +# +# 20 MB NimbleCSV 201-238 ms defk (copy) 249-257 ms 0.81-0.93x poor +# rusty_csv 57.6-57.7 ms: 3.5-4.1x NimbleCSV, 4.3-4.5x defk. +# The parts pass 1 4.7 ms, scan to the [i64] tape 28 ms, rows from +# (best of 5) it 314 ms; the copy 36 ms, its two splits 135 ms. +# Sweep 2k rows 0.87-0.89x, 20k 1.4-1.8x, 1M rows (100 MB) +# 2.4-2.65x: there NimbleCSV's garbage costs most. +# +# What it means: a kernel scans CSV at 0.5-0.7 GB/s, and Elixir then +# spends 4-11x the scan building the rows. A tape does not move the row in +# the wrapper table: for CSV (or JSON) to terms, use NimbleCSV, or rusty_csv +# when the speed matters; a NIF that builds sub-binaries in C does it +# 3.5-4x faster than Elixir does from any tape. Only the packed products +# (numbers, ids) are kernel jobs. + +Mix.install([ + {:defk, path: Path.expand("..", __DIR__)}, + {:nimble_csv, "~> 1.3"}, + {:rusty_csv, "~> 0.4"} +]) + +defmodule Tape.Source do + # The kernels, as a `use` block for Tape.Kernels, compiled with + # `elixir_mode: true` (native by default; `__defk_elixir__/2` is the + # agreement check's oracle). + defmacro __using__(_) do + quote location: :keep do + # Flags in a field's second tape word, above its length + @esc 1_099_511_627_776 + @eol 2_199_023_255_552 + + # Pass 1: an upper bound on the fields: one per ',' or '\n', plus one. + defk count_separators(s :: [u8]) :: i64 do + for c <- s, reduce: 1 do + acc -> if c == 44 or c == 10, do: wrap64(acc + 1), else: acc + end + end + + # Pass 2: RFC 4180 as NimbleCSV.RFC4180 reads it. Each field is two + # words: its start, and its length plus @esc (a quoted field with + # doubled quotes, to unescape) plus @eol (the last field of a row). + # Rows end in "\n" or "\r\n". A quote inside an unquoted field, or + # anything but a separator after a closing quote, raises ArgumentError. + # States: 0 row start, 1 unquoted field, 2 quoted field, 3 after a + # quote in a quoted field, 4 '\r' after a closing quote, 5 field + # start after ','. + defk scan(s :: [u8], cap :: i64) :: {[i64], i64} do + if length(s) >= @esc, do: raise(ArgumentError, "an input of 1 TiB or more") + tape = alloc(i64, 2 * cap) + n = length(s) + + {st, fs, qe, esc, k} = + for i <- 0..(n - 1)//1, reduce: {0, 0, 0, false, 0} do + {st, fs, qe, esc, k} -> + c = s[i] + + cond do + st == 2 -> + if c == 34, do: {3, fs, i, esc, k}, else: {2, fs, qe, esc, k} + + st == 3 and c == 34 -> + {2, fs, qe, true, k} + + st == 3 and c == 13 -> + {4, fs, qe, esc, k} + + st == 3 or st == 4 -> + if c != 44 and c != 10, do: raise(ArgumentError, "a byte after a closing quote") + if st == 4 and c != 10, do: raise(ArgumentError, "a '\\r' after a closing quote") + tape[2 * k] = fs + w = qe - fs + w = if esc, do: w + @esc, else: w + tape[2 * k + 1] = if c == 10, do: w + @eol, else: w + {if(c == 10, do: 0, else: 5), i + 1, qe, false, k + 1} + + c == 34 -> + if st == 1, do: raise(ArgumentError, "a quote inside an unquoted field") + {2, i + 1, qe, false, k} + + c == 44 -> + tape[2 * k] = if st == 1, do: fs, else: i + tape[2 * k + 1] = if st == 1, do: i - fs, else: 0 + {5, i + 1, qe, false, k + 1} + + c == 10 -> + f = if st == 1, do: fs, else: i + e = if i > f and s[i - 1] == 13, do: i - 1, else: i + tape[2 * k] = f + tape[2 * k + 1] = e - f + @eol + {0, i + 1, qe, false, k + 1} + + true -> + {1, if(st == 1, do: fs, else: i), qe, esc, k} + end + end + + if st == 2 or st == 4, do: raise(ArgumentError, "the input ends inside a quoted field") + + k = + cond do + st == 1 -> + tape[2 * k] = fs + tape[2 * k + 1] = n - fs + @eol + k + 1 + + st == 3 -> + tape[2 * k] = fs + w = qe - fs + tape[2 * k + 1] = if(esc, do: w + @esc, else: w) + @eol + k + 1 + + st == 5 -> + tape[2 * k] = n + tape[2 * k + 1] = @eol + k + 1 + + true -> + k + end + + {tape, k} + end + + # The other tape form: a cleaned copy. Separating commas become 31 + # (US), row ends 30 (RS, also after the last row), the quotes around + # a quoted field go and a doubled quote is written once; a "\r" + # before "\n" outside quotes goes. Returns {buffer, length}. A 30 or + # 31 byte in the input raises ArgumentError (this form's limit); the + # rest of the grammar is scan/2's. States as in scan/2. + defk normalize(s :: [u8]) :: {[u8], i64} do + n = length(s) + out = alloc(u8, n + 1) + + {st, j} = + for i <- 0..(n - 1)//1, reduce: {0, 0} do + {st, j} -> + c = s[i] + if c == 30 or c == 31, do: raise(ArgumentError, "a 30 or 31 byte in the input") + + cond do + st == 2 -> + if c == 34 do + {3, j} + else + out[j] = c + {2, j + 1} + end + + st == 3 and c == 34 -> + out[j] = 34 + {2, j + 1} + + st == 3 and c == 13 -> + {4, j} + + st == 3 or st == 4 -> + if c != 44 and c != 10, do: raise(ArgumentError, "a byte after a closing quote") + if st == 4 and c != 10, do: raise(ArgumentError, "a '\\r' after a closing quote") + out[j] = if c == 10, do: 30, else: 31 + {if(c == 10, do: 0, else: 5), j + 1} + + c == 34 -> + if st == 1, do: raise(ArgumentError, "a quote inside an unquoted field") + {2, j} + + c == 44 -> + out[j] = 31 + {5, j + 1} + + c == 10 -> + out[j] = 30 + {0, j + 1} + + c == 13 and i + 1 < n and s[i + 1] == 10 -> + {st, j} + + true -> + out[j] = c + {1, j + 1} + end + end + + if st == 2 or st == 4, do: raise(ArgumentError, "the input ends inside a quoted field") + closes = st == 1 or st == 3 or st == 5 + if closes, do: out[j] = 30 + {out, if(closes, do: j + 1, else: j)} + end + end + end +end + +defmodule Tape.Kernels do + use Defk, elixir_mode: true + use Tape.Source +end + +defmodule Tape.Build do + import Bitwise + + @len 1_099_511_627_775 + @esc 1_099_511_627_776 + @eol 2_199_023_255_552 + + # The rows NimbleCSV.RFC4180.parse_string(csv, skip_headers: false) + # returns: lists of sub-binaries of csv, a quoted field with doubled + # quotes unescaped into a new binary. + def rows(csv, {tape, k}), do: build(binary_part(tape, 0, 16 * k), csv, [], []) + + defp build(<>, csv, row, acc) do + f = binary_part(csv, s, w &&& @len) + f = if (w &&& @esc) != 0, do: :binary.replace(f, "\"\"", "\"", [:global]), else: f + + if (w &&& @eol) != 0, + do: build(rest, csv, [], [:lists.reverse([f | row]) | acc]), + else: build(rest, csv, [f | row], acc) + end + + defp build(<<>>, _csv, [], acc), do: :lists.reverse(acc) +end + +defmodule Tape.Calls do + def parse(csv), do: Tape.Build.rows(csv, scan(csv)) + def scan(csv), do: Tape.Kernels.scan(csv, Tape.Kernels.count_separators(csv)) + + # The cleaned copy, split by the :binary BIFs: rows at 30, fields at 31. + def parse_normalized(csv), do: split_normalized(Tape.Kernels.normalize(csv)) + + def split_normalized({_out, 0}), do: [] + + def split_normalized({out, j}) do + for row <- :binary.split(binary_part(out, 0, j - 1), <<30>>, [:global]), + do: :binary.split(row, <<31>>, [:global]) + end +end + +defmodule Tape.Data do + # A customer export: id, name, email, city, amount, signup date, and a + # free-text note that is quoted in a third of the rows (commas inside), + # with doubled quotes in a tenth and a line break in one row of 200. + @first ~w(Anna Bo Carlos Dana Emil Fatima Gustav Hana Ivan Jade Karl Lena) + @last ~w(Svensson Okafor Lindqvist Moreau Tanaka Novak Haddad Berg Costa Ito) + @city ["Stockholm", "Malmö", "Göteborg", "Uppsala", "Valletta", "London", "São Paulo", "Kraków"] + @words ~w(called about the invoice wants a refund new address prefers email VIP late payment) + + def csv(rows, seed \\ 1) do + :rand.seed(:exsss, {seed, 3, 5}) + + IO.iodata_to_binary( + for id <- 1..rows//1 do + fst = Enum.random(@first) + lst = Enum.random(@last) + + [ + Integer.to_string(id), + ?,, + fst, + " ", + lst, + ?,, + String.downcase(fst), + ?., + String.downcase(lst), + "@example.com,", + Enum.random(@city), + ?,, + :erlang.float_to_binary(:rand.uniform() * 5000, decimals: 2), + ?,, + "2026-", + pad(:rand.uniform(12)), + ?-, + pad(:rand.uniform(28)), + ?,, + note(), + "\r\n" + ] + end + ) + end + + defp pad(n) when n < 10, do: [?0, Integer.to_string(n)] + defp pad(n), do: Integer.to_string(n) + + defp note do + words = Enum.join(Enum.take_random(@words, :rand.uniform(6)), " ") + + case :rand.uniform(200) do + r when r <= 20 -> [?", words, ", said \"\"ok\"\"", ?"] + r when r <= 21 -> [?", words, "\nsecond line", ?"] + r when r <= 67 -> [?", words, ", ", words, ?"] + _ -> words + end + end +end + +alias Tape.{Calls, Data, Kernels} + +nimble = &NimbleCSV.RFC4180.parse_string(&1, skip_headers: false) +rusty = &RustyCSV.RFC4180.parse_string(&1, skip_headers: false) + +# ---- agreement ---- + +raises? = fn f -> + try do + f.() + false + rescue + _ -> true + end +end + +oracle = fn csv -> + Tape.Build.rows(csv, Kernels.__defk_elixir__(:scan, [csv, Kernels.count_separators(csv)])) +end + +cases = + [ + "", "a", "a,b", "a,b\n", "a,b\r\n", "a,b\n\nc,d\n", "a,\n,b\n", ",", ",\n", "\n", "\r\n", + "\"a,1\",\"x\"\"y\",\"\"\n", "\"q\"", "\"q\"\r\n", "\"multi\nline\",x\n", "\"\"\"\"", "a\r", "x,\"y\"", + Data.csv(3000, 9) + ] ++ for(r <- 1..30, do: Data.csv(r, r)) + +agree = Enum.all?(cases, fn c -> (p = Calls.parse(c)) == nimble.(c) and p == oracle.(c) and p == Calls.parse_normalized(c) and p == Calls.split_normalized(Kernels.__defk_elixir__(:normalize, [c])) end) +bad = ["a\"b,c\n", "\"a\"b,c\n", "\"open\n", "\"open", "\"a\"\rb", "x,\"y\"z"] + +bad_ok = + Enum.all?(bad, fn c -> + raises?.(fn -> Calls.parse(c) end) and raises?.(fn -> oracle.(c) end) and raises?.(fn -> nimble.(c) end) and raises?.(fn -> Calls.parse_normalized(c) end) + end) + +csv = Data.csv(200_000) +rows = nimble.(csv) +rusty_agree = rusty.(csv) == rows + +IO.puts(""" +agreement: tape == NimbleCSV and Elixir mode on #{length(cases)} inputs #{agree}, \ +bad input raises on every side #{bad_ok} (#{length(bad)} cases), rusty_csv agrees #{rusty_agree} +""") + +# As Defk.Bench.report/1 does: a full sweep and a minor GC before each +# timed run, so every run starts from a small young heap. Without them a +# run reuses the heap the previous one grew, and term building looks +# 1.5-2x cheaper than a caller sees it. +best = fn f, runs -> + for _ <- 1..runs do + :erlang.garbage_collect() + :erlang.garbage_collect(self(), type: :minor) + elem(:timer.tc(f), 0) + end + |> Enum.min() + |> Kernel./(1000) + |> Float.round(3) +end + +tape = Calls.scan(csv) +norm = Kernels.normalize(csv) + +IO.puts("csv, #{Float.round(byte_size(csv) / 1_000_000, 1)} MB (#{length(rows)} rows x 7 fields), best of 5:") + +for {label, f} <- [ + {"NimbleCSV.RFC4180.parse_string", fn -> nimble.(csv) end}, + {"rusty_csv parse_string (Rust NIF)", fn -> rusty.(csv) end}, + {"defk pass 1 (count separators)", fn -> Kernels.count_separators(csv) end}, + {"defk pass 2 (scan to tape)", fn -> Kernels.scan(csv, elem(tape, 1) + 1) end}, + {"Elixir rows from the tape", fn -> Tape.Build.rows(csv, tape) end}, + {"tape end to end", fn -> Calls.parse(csv) end}, + {"defk normalize (cleaned copy)", fn -> Kernels.normalize(csv) end}, + {"Elixir rows from the copy (2 splits)", fn -> Calls.split_normalized(norm) end}, + {"cleaned copy end to end", fn -> Calls.parse_normalized(csv) end} + ] do + IO.puts(" #{String.pad_trailing(label, 36)} #{best.(f, 5)} ms") +end + +# ---- the verdict: the cleaned copy (the faster tape form) against +# NimbleCSV, with rusty_csv's NIF beside it ---- + +sweep = fn rows -> + c = Data.csv(rows, 17) + {fn -> nimble.(c) end, fn -> Calls.parse_normalized(c) end} +end + +Defk.Bench.report( + name: "tape_csv", + expected: :poor, + baseline: :first_pass, + runs: 15, + realistic_size: "20 MB: a 200k-row customer export", + elixir: fn -> nimble.(csv) end, + defk: fn -> Calls.parse_normalized(csv) end, + native: {fn -> rusty.(csv) end, "rusty_csv 0.4.6"}, + check: fn -> agree and bad_ok and rusty_agree and Calls.parse_normalized(csv) == rows end, + sweep: for(r <- [2_000, 20_000, 1_000_000], do: Tuple.insert_at(sweep.(r), 0, "#{r} rows")), + sweep_runs: 3, + reason: + "the kernel scans 20 MB in 28-36 ms, but Elixir still builds every row and field, which costs more than NimbleCSV's whole parse; rusty_csv builds the same sub-binaries in C, 3.5x faster" +) From 71e2977712249a7306ca0a8145883438845093f6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20L=C3=BCning?= Date: Tue, 20 Oct 2026 15:40:37 +0200 Subject: [PATCH 2/3] defk-9p6.30: time each parse in a fresh process; the cleaned copy is 2.2x the best Elixir In one long-lived process the verdict moved from 0.81x to 2.08x run to run with the heap the earlier runs left. A fresh process per parse is a request handler's case and repeats to 2%. Two more Elixir variants (E2: :binary.split lines and fields, quoted lines walked with :binary.match; E3: quoted pieces joined); E2 is the baseline, 5-7% faster than NimbleCSV. The cleaned copy: 2.23x, good (pre-registered marginal). rusty_csv is 2.6x faster than the copy. --- docs/performance.md | 49 +++---- examples/README.md | 14 +- examples/_tables.exs | 14 +- examples/tape_csv.exs | 292 ++++++++++++++++++++++++++++++++++-------- 4 files changed, 281 insertions(+), 88 deletions(-) diff --git a/docs/performance.md b/docs/performance.md index 94efd818..daa1febb 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -2906,32 +2906,35 @@ output array would). - **Source:** [`bench/port_roundtrip.exs`](https://github.com/DanielLuning/defk/blob/main/bench/port_roundtrip.exs), [`bench/port_echo.c`](https://github.com/DanielLuning/defk/blob/main/bench/port_echo.c), `defk-7dm`. - **Status:** one quiet run; rerun with `defk-sgw.4`'s snapshot if the verdict cites it. -### 7.11 CSV to terms: a kernel tape plus Elixir row building loses to NimbleCSV; rusty_csv builds the sub-binaries in C, 3.5-4x faster - -A kernel can scan the CSV and hand Elixir flat events, but Elixir then -builds every row and field, and that half costs more than NimbleCSV's -whole parse. A NIF that makes the sub-binaries itself is the fast route. - -- **Numbers:** 20 MB, a 200k-row export with quoted fields: NimbleCSV - 201-238 ms; rusty_csv 0.4.6 57.6-57.7 ms; the kernel's cleaned copy - (separators as bytes 30 and 31, split twice by `:binary.split/3`) - 249-257 ms end to end, 0.81-0.93x. Its parts: the copy 36 ms, the two - splits 135 ms. An `[i64]` tape of start and length per field: scan 28 - ms, rows built from it with binary matching and `binary_part/3` 314 ms. - Walking 1.4M tape entries without building anything took 39 ms. At 100 - MB the copy is 2.4-2.65x NimbleCSV, whose garbage costs most there. -- **Heap state matters for term building:** without a GC between timed - runs, each run reused the heap the last one grew, and the rows from the - tape took 83-92 ms instead of 314 ms (NimbleCSV 134-138 instead of - 201-238). `Defk.Bench.report/1` collects before each run; variant loops - in a script must do the same. +### 7.11 CSV to terms: a kernel that cleans the separators for `:binary.split` is 2.2x the best Elixir; rusty_csv builds the sub-binaries in C, 2.6x faster again + +A kernel cannot build terms, but it can make the input easy for the BIFs: +a cleaned copy with byte 31 between fields and byte 30 after each row, +quotes removed, that two `:binary.split/3` passes turn into NimbleCSV's +rows. A tape of field events walked in Elixir is slower: reading the tape +costs as much as the splits. + +- **Numbers** (20 MB, a 200k-row export with quoted fields; a fresh process + per parse): NimbleCSV 308-315 ms; the best Elixir (lines by + `:binary.split`, unquoted lines split on commas, quoted lines walked with + `:binary.match`) 288-299 ms; the cleaned copy 131-142 ms end to end + (2.23x; the copy 36 ms, the splits 125-131 ms); an `[i64]` tape of start + and length per field 152-158 ms (scan 28 ms, rows from it 119-127 ms; + only walking its 1.4M entries took 39 ms); rusty_csv 0.4.6 49.6-53.5 ms. +- **Heap state decides term-building benchmarks:** timed in one long-lived + process, best of 15, the same code gave NimbleCSV 134-358 ms and the copy + 96-264 ms, 0.81x to 2.08x between runs, with or without a GC before each + run. A fresh process per run grows its heap from nothing every time; the + spread was 2% (median against best). - **What it means:** the 13x gap between scanning and term creation that `string-workloads.md` measured holds for Elixir, not for C: rusty_csv's - sub-binaries cost little. For CSV or JSON to terms, use NimbleCSV or - Jason, or the NIF when the speed matters. -- **Conditions:** M-series Mac, OTP 29, Elixir 1.20.2, two runs. + sub-binaries cost little. For CSV to rows, rusty_csv is the fastest; a + kernel's cleaned copy is a middle option without another NIF, and yields + on a normal scheduler. JSON to terms is not measured. +- **Conditions:** M-series Mac, OTP 29, Elixir 1.20.2; three runs with a + fresh process, five before in a long-lived one. - **Source:** [`tape_csv.exs`](https://github.com/DanielLuning/defk/blob/main/examples/tape_csv.exs), `defk-9p6.30`. -- **Status:** provisional. +- **Status:** provisional until the stored run. --- diff --git a/examples/README.md b/examples/README.md index 52303795..38ebd05a 100644 --- a/examples/README.md +++ b/examples/README.md @@ -194,12 +194,12 @@ has them in full. that on CSV, and about 4× on JSON against Jason, OTP's `:json` and the same parser. - [`tape_csv.exs`](tape_csv.exs): CSV to NimbleCSV's rows of binaries - through a kernel tape (field events, or a cleaned copy split by - `:binary.split`). The scan takes 28–36 ms of 20 MB; building the rows in - Elixir costs more than NimbleCSV's whole parse, so the tape is poor - (0.81–0.93×). rusty_csv's NIF builds the same sub-binaries in C, 3.5–4× - faster than NimbleCSV. Benchmarks of term building must start each run - from a collected heap: without it the rows looked 1.5–4× cheaper. + through a kernel tape. An `[i64]` tape of field events loses to a + cleaned copy (separators as bytes 30 and 31) that two `:binary.split` + passes turn into rows. In a fresh process per parse the copy is 2.2× the + best Elixir; rusty_csv's NIF builds the same sub-binaries in C, 2.6× + faster again. Timed in one long-lived process, the verdict moved from + 0.8× to 2.1× run to run with the heap the earlier runs left. - [`lz4_snappy_decode.exs`](lz4_snappy_decode.exs): Snappy and LZ4 block decode from scratch on untrusted input, against snappyer, lz4_erl and nimble_lz4. Against the NIFs as installed from Hex (built with no `-O`) @@ -305,7 +305,7 @@ Nothing here is excluded on principle: each row names the condition that would r | Job | Wrapped today | Why not from scratch | What would change it | |---|---|---|---| | Signing, key exchange, secret-dependent crypto (secp256k1, ed25519, X25519, AEAD) | libsodium, secp256k1, ring via Rustler; `:crypto` | The one rule says nothing about timing: clang `-O3` may turn a select into a branch or a table lookup into a secret-dependent access, and a process dump or yielded frame holds key material. Public-input verification is allowed but gains little. | A constant-time mode: a restricted subset checked at compile time, codegen verified with dudect or ctgrind in CI, frame zeroisation, and an external audit. Until then, `:crypto`. | -| Parsers that produce trees or term streams (HTML, Markdown, tree-sitter, JSON/CSV to terms, protobuf, msgpack) | html5ever, comrak, tree-sitter, jsont, rusty_csv, protox | The product is BEAM terms and building them is the cost: `docs/research/string-workloads.md` measured a 13x gap between scanning and term creation. [`tape_csv`](tape_csv.exs) measured the tape route: a kernel scans 20 MB of CSV in 28–36 ms, and Elixir then takes 135–314 ms to build the rows from its events, so the whole is 0.81–0.93× NimbleCSV (poor). rusty_csv builds the same sub-binaries in C, 3.5–4× faster than NimbleCSV: here the NIF builds terms far cheaper than Elixir does. A kernel cannot build terms. | Closed with evidence for a tape (`tape_csv`). It would take a kernel that builds the sub-binaries and lists itself, which the kernel subset leaves out. BPE tokenisation to `[i64]` ids is the exception: its product is packed, and so is a numeric CSV or JSON number array loaded into f64s (`numeric_loader`, good: about 2.7× and 4×). | +| Parsers that produce trees or term streams (HTML, Markdown, tree-sitter, JSON/CSV to terms, protobuf, msgpack) | html5ever, comrak, tree-sitter, jsont, rusty_csv, protox | The product is BEAM terms and building them is the cost: `docs/research/string-workloads.md` measured a 13x gap between scanning and term creation. [`tape_csv`](tape_csv.exs) measured the tape route for CSV: a kernel cleans 20 MB in 36 ms, and two `:binary.split` passes then take 125–131 ms to build the rows, 2.2× the best Elixir in a fresh process per parse (in one long-lived process 0.8–2.1×, with the heap). rusty_csv builds the same sub-binaries in C, 2.6× faster again: a NIF builds terms far cheaper than Elixir does. A kernel cannot build terms. | Reopened narrowly for CSV to rows (`tape_csv`, good) where an extra NIF is not wanted; rusty_csv stays the fastest. JSON to terms is not measured. BPE tokenisation to `[i64]` ids is the exception: its product is packed, and so is a numeric CSV or JSON number array loaded into f64s (`numeric_loader`, good: about 2.7× and 4×). | | Regular expressions | `:re` (PCRE in C), Rust regex via Rustler | `:re` is already native and compiled once per pattern; a kernel cannot take a pattern as data without becoming an interpreter, and fixed-string search is `:binary.match`. | A specific pattern class with a known kernel form (glob, or a simple byte DFA compiled at Elixir compile time) and a benchmark showing `:re` loses on it. | | Large codecs and encoders (brotli, zstd encode, audio and video, PDF/typst rendering) | brotli, zstd, membrane_* FFmpeg/opus bindings, typst | The cost is spec surface and tuning, not the loop: 10^4-10^5 lines with SIMD everywhere. A kernel would be a worse implementation of a solved problem. | A narrow, named need: one compression level with fixed tables (zstd level 1 is the lz4-encode ticket plus FSE) or a decoder-only requirement for untrusted input. Each reopens one row, not the category. | | Systems: databases, dataframes, tensor compilers (SQLite, DuckDB, Polars/Explorer, XLA/EXLA, LibTorch) | exqlite, duckdbex, explorer, exla, torchx | The value is the library: query planners, columnar engines, autodiff, device backends. defk is a kernel, not a system. | Interop, not rewriting: zero-copy views over their packed buffers so a per-element loop beside them runs as a kernel. `docs/research/nx-boundary.md` says where that line is; file an interop ticket when a workload asks. | diff --git a/examples/_tables.exs b/examples/_tables.exs index 95085e81..8bb9b9a3 100644 --- a/examples/_tables.exs +++ b/examples/_tables.exs @@ -161,9 +161,9 @@ line: "{speedup:sort_to_list} when the caller needs a list back" }, "tape_csv" => %{ - job: "CSV of 20 MB to NimbleCSV's rows, through a kernel tape", + job: "CSV of 20 MB to NimbleCSV's rows, a fresh process per parse", native: "{native} rusty_csv", - line: "the scan is fast; building the rows in Elixir costs more than NimbleCSV's whole parse" + line: "in a long-lived process 0.8–2.1× run to run; an [i64] tape of field events is slower than the cleaned copy" }, "utf8_valid" => %{ job: "validate 16 MB of UTF-8", @@ -232,6 +232,11 @@ "a numeric CSV or a JSON array of numbers straight into packed f64s (NimbleCSV and Jason stay for anything else)", examples: ["numeric_loader"] }, + %{ + text: + "CSV to NimbleCSV's rows through a kernel that cleans the separators for `:binary.split` (rusty_csv's NIF is faster still)", + examples: ["tape_csv"] + }, %{ text: "a fuzzy-search web service, Myers bit-parallel on both sides: `/health` latency stays at Elixir's while a blocking NIF raises it", @@ -252,11 +257,6 @@ "anything `:crypto` does (a SHA-256 chain) and memory-hard password hashing (Argon2: the C runs its lanes on threads)", examples: ["hash_chain", "argon2"] }, - %{ - text: - "CSV to rows of binaries through a kernel tape: building the terms is the cost (rusty_csv's NIF builds them in C, 3.5× NimbleCSV)", - examples: ["tape_csv"] - }, %{text: "one kernel call per element", examples: ["poor_tiny_calls"]}, %{ text: "data that arrives as lists: packing costs more than the whole `Enum.sum`", diff --git a/examples/tape_csv.exs b/examples/tape_csv.exs index aeb805db..2ce5375c 100644 --- a/examples/tape_csv.exs +++ b/examples/tape_csv.exs @@ -12,7 +12,7 @@ # elixir examples/tape_csv.exs # # Pre-registered: marginal (the research: Elixir still builds every -# sub-binary). Measured: poor, and the expected verdict below says so. +# sub-binary). Measured: good, and the expected verdict below says so. # # Two tape forms, both from the same RFC 4180 grammar as NimbleCSV # (quoted fields with commas, doubled quotes and line breaks; \n or \r\n): @@ -27,11 +27,13 @@ # the input raise ArgumentError (this form's limit). # The copy is the faster form and the verdict's defk side. # -# Elixir baseline: first pass, NimbleCSV.RFC4180 (which already splits -# unquoted runs with :binary.split). Not tried: a hand-written Elixir -# parser. With defk behind NimbleCSV, a faster Elixir side only widens the -# gap. Native: rusty_csv 0.4.6 (Rust NIF, precompiled; SIMD scan, then -# sub-binary terms built in C on a dirty scheduler). +# Elixir baseline: optimised. E1 NimbleCSV.RFC4180 (it already splits +# unquoted runs with :binary.split); E2 lines by :binary.split, a line +# without a quote split on commas, a quoted line walked with :binary.match +# (the baseline: 5-7% faster than E1); E3 E2 with quoted lines split on +# commas and the pieces joined (no gain). Native: rusty_csv 0.4.6 (Rust +# NIF, precompiled; SIMD scan, then sub-binary terms built in C on a dirty +# scheduler). # # Agreement: both tape forms, their Elixir mode (`elixir_mode: true`, # Tape.Kernels.__defk_elixir__/2) and rusty_csv return exactly NimbleCSV's @@ -44,28 +46,34 @@ # amount, date, a note quoted in a third of the rows), with a sweep of 2k, # 20k and 1M rows (100 MB). # -# Timing: every run starts from a collected heap (Defk.Bench.report/1 does -# this, and so do the variant lines). Without the GC between runs, each run -# reuses the heap the previous one grew, and building the terms looked -# 1.5-4x cheaper than a caller sees it (the [i64] tape's rows 83-92 ms -# instead of 314 ms). +# Timing: every timed parse runs in a fresh process (spawn, parse, exit), +# as a request handler or a Task per file would. Measured first in one +# long-lived process, best of 15, the answer depended on the heap the +# earlier runs left: with a GC before each run (as Defk.Bench.report/1 +# does) NimbleCSV 201-358 ms and the copy 172-257 ms, 0.81x to 2.08x +# between runs; with no GC between runs NimbleCSV 134-138 ms and the copy +# 96-113 ms. In a fresh process every run grows its heap from nothing, +# and the spread is 2% (median against best). # -# Measured (M-series Mac, OTP 29, Elixir 1.20, 2026-10-20, best of 15; two -# runs, provisional until the stored run): +# Measured (M-series Mac, OTP 29, Elixir 1.20, 2026-10-20, fresh process +# per run; the report best of 15, the variant lines best of 5; three runs +# with E2, provisional until the stored run): # -# 20 MB NimbleCSV 201-238 ms defk (copy) 249-257 ms 0.81-0.93x poor -# rusty_csv 57.6-57.7 ms: 3.5-4.1x NimbleCSV, 4.3-4.5x defk. -# The parts pass 1 4.7 ms, scan to the [i64] tape 28 ms, rows from -# (best of 5) it 314 ms; the copy 36 ms, its two splits 135 ms. -# Sweep 2k rows 0.87-0.89x, 20k 1.4-1.8x, 1M rows (100 MB) -# 2.4-2.65x: there NimbleCSV's garbage costs most. +# 20 MB E1 308-315 ms, E2 288-299 ms, E3 300-305 ms; +# the copy 130.7-141.9 ms 2.23x over E2 good +# rusty_csv 49.6-53.5 ms: 5.8x E2, 2.6x the copy. +# The parts pass 1 4.8-5.1 ms, scan to the [i64] tape 28 ms, rows +# (best of 5) from it 119-127 ms (the tape 152-158 ms end to end); +# the copy 36 ms, its two splits 125-131 ms. +# Sweep 2k rows 1.7x, 20k 1.7x, 1M rows (100 MB) 2.5x. # # What it means: a kernel scans CSV at 0.5-0.7 GB/s, and Elixir then -# spends 4-11x the scan building the rows. A tape does not move the row in -# the wrapper table: for CSV (or JSON) to terms, use NimbleCSV, or rusty_csv -# when the speed matters; a NIF that builds sub-binaries in C does it -# 3.5-4x faster than Elixir does from any tape. Only the packed products -# (numbers, ids) are kernel jobs. +# spends 3.5-4.5x the scan building the rows. The cleaned copy, split by +# the BIFs, beats the best Elixir by about 2.2x in a fresh process, so a +# tape does reopen the wrapper table's row, narrowly. rusty_csv builds the +# same sub-binaries in C, 2.6x faster again, on a dirty scheduler: where +# the speed matters, it is still the choice. The kernel route needs no +# extra NIF, and its call yields on a normal scheduler. Mix.install([ {:defk, path: Path.expand("..", __DIR__)}, @@ -120,7 +128,10 @@ defmodule Tape.Source do st == 3 or st == 4 -> if c != 44 and c != 10, do: raise(ArgumentError, "a byte after a closing quote") - if st == 4 and c != 10, do: raise(ArgumentError, "a '\\r' after a closing quote") + + if st == 4 and c != 10, + do: raise(ArgumentError, "a '\\r' after a closing quote") + tape[2 * k] = fs w = qe - fs w = if esc, do: w + @esc, else: w @@ -209,7 +220,10 @@ defmodule Tape.Source do st == 3 or st == 4 -> if c != 44 and c != 10, do: raise(ArgumentError, "a byte after a closing quote") - if st == 4 and c != 10, do: raise(ArgumentError, "a '\\r' after a closing quote") + + if st == 4 and c != 10, + do: raise(ArgumentError, "a '\\r' after a closing quote") + out[j] = if c == 10, do: 30, else: 31 {if(c == 10, do: 0, else: 5), j + 1} @@ -287,6 +301,149 @@ defmodule Tape.Calls do end end +defmodule Tape.Elixir do + # The :binary BIFs do the scanning. Lines by :binary.split; a line + # without a quote is split on commas. A line with one is joined with the + # next while a quote is open, then E2 walks it field by field with + # :binary.match, and E3 splits it on commas and joins the pieces of each + # quoted field. The same rows as NimbleCSV.RFC4180. + def parse(csv), do: parse(csv, :walk) + def parse_merge(csv), do: parse(csv, :merge) + + defp parse("", _mode), do: [] + + # The last line is open when the input does not end in "\n": NimbleCSV + # keeps a "\r" there. + defp parse(csv, mode) do + lines = :binary.split(csv, "\n", [:global]) + + if :binary.last(csv) == ?\n, + do: rows(:lists.droplast(lines), true, mode, []), + else: rows(lines, false, mode, []) + end + + defp rows([], _closed, _mode, acc), do: :lists.reverse(acc) + + defp rows([l | rest], closed, mode, acc) do + case :binary.match(l, "\"") do + :nomatch -> + rows(rest, closed, mode, [:binary.split(strip_cr(l, rest, closed), ",", [:global]) | acc]) + + _ -> + record(l, rest, closed, mode, acc) + end + end + + # join lines until the record's quotes balance + defp record(l, rest, closed, mode, acc) do + if rem(length(:binary.matches(l, "\"")), 2) == 0 do + r = strip_cr(l, rest, closed) + + row = + if mode == :walk, do: fields(r, 0, []), else: merge(:binary.split(r, ",", [:global]), []) + + rows(rest, closed, mode, [row | acc]) + else + case rest do + [next | rest] -> record(<>, rest, closed, mode, acc) + [] -> raise ArgumentError, "the input ends inside a quoted field" + end + end + end + + # E3: pieces between commas; a piece that opens a quote takes the next + # pieces until its quotes balance and it ends in a quote + defp merge([], acc), do: :lists.reverse(acc) + defp merge([<> = p | rest], acc), do: collect(p, rest, acc) + + defp merge([p | rest], acc) do + if :binary.match(p, "\"") != :nomatch, + do: raise(ArgumentError, "a quote inside an unquoted field") + + merge(rest, [p | acc]) + end + + defp collect(buf, rest, acc) do + n = byte_size(buf) + + if n >= 2 and :binary.last(buf) == ?" and rem(length(:binary.matches(buf, "\"")), 2) == 0 do + inner = binary_part(buf, 1, n - 2) + + case :binary.match(inner, "\"") do + :nomatch -> + merge(rest, [inner | acc]) + + _ -> + # a quote that is not doubled ends the field early: NimbleCSV raises + if :binary.match(:binary.replace(inner, "\"\"", "", [:global]), "\"") != :nomatch, + do: raise(ArgumentError, "a byte after a closing quote") + + merge(rest, [:binary.replace(inner, "\"\"", "\"", [:global]) | acc]) + end + else + case rest do + [p | rest] -> collect(<>, rest, acc) + [] -> raise ArgumentError, "a byte after a closing quote" + end + end + end + + defp strip_cr(l, [], false), do: l + + defp strip_cr(l, _rest, _closed) do + n = byte_size(l) + if n > 0 and :binary.at(l, n - 1) == ?\r, do: binary_part(l, 0, n - 1), else: l + end + + defp fields(r, p, acc) do + n = byte_size(r) + + if p < n and :binary.at(r, p) == ?" do + {f, q} = quoted(r, p + 1, p + 1, false) + + cond do + q == n -> :lists.reverse([f | acc]) + :binary.at(r, q) == ?, -> next(r, q + 1, [f | acc]) + true -> raise ArgumentError, "a byte after a closing quote" + end + else + case :binary.match(r, ",", scope: {p, n - p}) do + {c, 1} -> next(r, c + 1, [unquoted(r, p, c - p) | acc]) + :nomatch -> :lists.reverse([unquoted(r, p, n - p) | acc]) + end + end + end + + # after a ',' a field always follows, empty at the end of the record + defp next(r, p, acc) when p == byte_size(r), do: :lists.reverse(["" | acc]) + defp next(r, p, acc), do: fields(r, p, acc) + + defp unquoted(r, p, len) do + f = binary_part(r, p, len) + + if :binary.match(f, "\"") != :nomatch, + do: raise(ArgumentError, "a quote inside an unquoted field") + + f + end + + # from s, the field's first byte; returns {field, index after the closing quote} + defp quoted(r, s, p, esc) do + case :binary.match(r, "\"", scope: {p, byte_size(r) - p}) do + :nomatch -> + raise ArgumentError, "the input ends inside a quoted field" + + {q, 1} -> + if q + 1 < byte_size(r) and :binary.at(r, q + 1) == ?" do + quoted(r, s, q + 2, true) + else + f = binary_part(r, s, q - s) + {if(esc, do: :binary.replace(f, "\"\"", "\"", [:global]), else: f), q + 1} + end + end + end +end + defmodule Tape.Data do # A customer export: id, name, email, city, amount, signup date, and a # free-text note that is quoted in a third of the rows (commas inside), @@ -368,17 +525,41 @@ end cases = [ - "", "a", "a,b", "a,b\n", "a,b\r\n", "a,b\n\nc,d\n", "a,\n,b\n", ",", ",\n", "\n", "\r\n", - "\"a,1\",\"x\"\"y\",\"\"\n", "\"q\"", "\"q\"\r\n", "\"multi\nline\",x\n", "\"\"\"\"", "a\r", "x,\"y\"", + "", + "a", + "a,b", + "a,b\n", + "a,b\r\n", + "a,b\n\nc,d\n", + "a,\n,b\n", + ",", + ",\n", + "\n", + "\r\n", + "\"a,1\",\"x\"\"y\",\"\"\n", + "\"q\"", + "\"q\"\r\n", + "\"multi\nline\",x\n", + "\"\"\"\"", + "a\r", + "x,\"y\"", Data.csv(3000, 9) ] ++ for(r <- 1..30, do: Data.csv(r, r)) -agree = Enum.all?(cases, fn c -> (p = Calls.parse(c)) == nimble.(c) and p == oracle.(c) and p == Calls.parse_normalized(c) and p == Calls.split_normalized(Kernels.__defk_elixir__(:normalize, [c])) end) +agree = + Enum.all?(cases, fn c -> + (p = Calls.parse(c)) == nimble.(c) and p == oracle.(c) and p == Calls.parse_normalized(c) and + p == Calls.split_normalized(Kernels.__defk_elixir__(:normalize, [c])) and + p == Tape.Elixir.parse(c) and p == Tape.Elixir.parse_merge(c) + end) + bad = ["a\"b,c\n", "\"a\"b,c\n", "\"open\n", "\"open", "\"a\"\rb", "x,\"y\"z"] bad_ok = Enum.all?(bad, fn c -> - raises?.(fn -> Calls.parse(c) end) and raises?.(fn -> oracle.(c) end) and raises?.(fn -> nimble.(c) end) and raises?.(fn -> Calls.parse_normalized(c) end) + raises?.(fn -> Calls.parse(c) end) and raises?.(fn -> oracle.(c) end) and + raises?.(fn -> nimble.(c) end) and raises?.(fn -> Calls.parse_normalized(c) end) and + raises?.(fn -> Tape.Elixir.parse(c) end) and raises?.(fn -> Tape.Elixir.parse_merge(c) end) end) csv = Data.csv(200_000) @@ -390,28 +571,37 @@ agreement: tape == NimbleCSV and Elixir mode on #{length(cases)} inputs #{agree} bad input raises on every side #{bad_ok} (#{length(bad)} cases), rusty_csv agrees #{rusty_agree} """) -# As Defk.Bench.report/1 does: a full sweep and a minor GC before each -# timed run, so every run starts from a small young heap. Without them a -# run reuses the heap the previous one grew, and term building looks -# 1.5-2x cheaper than a caller sees it. -best = fn f, runs -> - for _ <- 1..runs do - :erlang.garbage_collect() - :erlang.garbage_collect(self(), type: :minor) - elem(:timer.tc(f), 0) +# Every timed parse runs in a fresh process, as a request handler would, +# and the time includes its exit. Building 1.4M terms in one long-lived +# process gave answers that depended on the heap the earlier runs left: +# NimbleCSV 134-358 ms, the copy 96-264 ms, the verdict 0.8x to 2.1x. +fresh = fn f -> + fn -> + {_pid, ref} = spawn_monitor(fn -> f.() end) + + receive do + {:DOWN, ^ref, :process, _, :normal} -> :ok + {:DOWN, ^ref, :process, _, reason} -> exit(reason) + end end - |> Enum.min() - |> Kernel./(1000) - |> Float.round(3) +end + +best = fn f, runs -> + g = fresh.(f) + for(_ <- 1..runs, do: elem(:timer.tc(g), 0)) |> Enum.min() |> Kernel./(1000) |> Float.round(3) end tape = Calls.scan(csv) norm = Kernels.normalize(csv) -IO.puts("csv, #{Float.round(byte_size(csv) / 1_000_000, 1)} MB (#{length(rows)} rows x 7 fields), best of 5:") +IO.puts( + "csv, #{Float.round(byte_size(csv) / 1_000_000, 1)} MB (#{length(rows)} rows x 7 fields), best of 5:" +) for {label, f} <- [ - {"NimbleCSV.RFC4180.parse_string", fn -> nimble.(csv) end}, + {"E1 NimbleCSV.RFC4180.parse_string", fn -> nimble.(csv) end}, + {"E2 :binary.split, quoted lines walked", fn -> Tape.Elixir.parse(csv) end}, + {"E3 :binary.split, quoted pieces joined", fn -> Tape.Elixir.parse_merge(csv) end}, {"rusty_csv parse_string (Rust NIF)", fn -> rusty.(csv) end}, {"defk pass 1 (count separators)", fn -> Kernels.count_separators(csv) end}, {"defk pass 2 (scan to tape)", fn -> Kernels.scan(csv, elem(tape, 1) + 1) end}, @@ -429,21 +619,21 @@ end sweep = fn rows -> c = Data.csv(rows, 17) - {fn -> nimble.(c) end, fn -> Calls.parse_normalized(c) end} + {fresh.(fn -> Tape.Elixir.parse(c) end), fresh.(fn -> Calls.parse_normalized(c) end)} end Defk.Bench.report( name: "tape_csv", - expected: :poor, - baseline: :first_pass, + expected: :good, + baseline: :optimized, runs: 15, realistic_size: "20 MB: a 200k-row customer export", - elixir: fn -> nimble.(csv) end, - defk: fn -> Calls.parse_normalized(csv) end, - native: {fn -> rusty.(csv) end, "rusty_csv 0.4.6"}, + elixir: fresh.(fn -> Tape.Elixir.parse(csv) end), + defk: fresh.(fn -> Calls.parse_normalized(csv) end), + native: {fresh.(fn -> rusty.(csv) end), "rusty_csv 0.4.6"}, check: fn -> agree and bad_ok and rusty_agree and Calls.parse_normalized(csv) == rows end, sweep: for(r <- [2_000, 20_000, 1_000_000], do: Tuple.insert_at(sweep.(r), 0, "#{r} rows")), sweep_runs: 3, reason: - "the kernel scans 20 MB in 28-36 ms, but Elixir still builds every row and field, which costs more than NimbleCSV's whole parse; rusty_csv builds the same sub-binaries in C, 3.5x faster" + "the kernel cleans 20 MB in 36 ms and two :binary.split passes build the rows, with less garbage than a parser in Elixir; rusty_csv builds them in C, 2.7x faster still" ) From 6ff17380159aacdb9ad502cd11abb913ca9c5059 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20L=C3=BCning?= Date: Tue, 20 Oct 2026 15:42:16 +0200 Subject: [PATCH 3/3] defk-9p6.30: store tape_csv's quiet row at 71e29777; ranges over four runs mix defk.examples --bench --only tape_csv --snapshot --fresh from a clean tree: quiet, 2.09x over E2, rusty_csv 49.5 ms. The header keeps its line count, so the stamped C is the committed kernels'. --- README.md | 1 + docs/performance.md | 12 ++++++------ examples/README.md | 9 +++++---- examples/scoreboard.json | 1 + examples/tape_csv.exs | 14 +++++++------- 5 files changed, 20 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index b30c64d1..6c9644ef 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,7 @@ Speedups over an optimised Elixir baseline, measured end to end on Apple M4 Pro | ✅ | base64, hex and base58 codecs (4.2× the fast64 NIF); Snappy and LZ4 block decode on untrusted input | 5.1–6.1× | | ✅ | sorting packed keys, nearest of 500k locations, Monte Carlo slot simulation with `:rand` bit for bit, Gorilla encoding | 2.4–6.0× | | ✅ | a numeric CSV or a JSON array of numbers straight into packed f64s (NimbleCSV and Jason stay for anything else) | 2.7× | +| ✅ | CSV to NimbleCSV's rows through a kernel that cleans the separators for `:binary.split` (rusty_csv's NIF is faster still) | 2.1× | | ✅ | a fuzzy-search web service, Myers bit-parallel on both sides: `/health` latency stays at Elixir's while a blocking NIF raises it | [its own measurement](https://github.com/DanielLuning/defk/blob/main/examples/fuzzy_search/README.md) | | 🟡 | a dictionary-wide Levenshtein batch over all schedulers: Myers bit-parallel in Elixir is close | 1.5× | | 🟡 | small WebSocket frames, UTF-8 validation, substring search: the BIFs are native already | 1.0–1.5× | diff --git a/docs/performance.md b/docs/performance.md index daa1febb..0ca080f4 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -2906,7 +2906,7 @@ output array would). - **Source:** [`bench/port_roundtrip.exs`](https://github.com/DanielLuning/defk/blob/main/bench/port_roundtrip.exs), [`bench/port_echo.c`](https://github.com/DanielLuning/defk/blob/main/bench/port_echo.c), `defk-7dm`. - **Status:** one quiet run; rerun with `defk-sgw.4`'s snapshot if the verdict cites it. -### 7.11 CSV to terms: a kernel that cleans the separators for `:binary.split` is 2.2x the best Elixir; rusty_csv builds the sub-binaries in C, 2.6x faster again +### 7.11 CSV to terms: a kernel that cleans the separators for `:binary.split` is 2.1-2.2x the best Elixir; rusty_csv builds the sub-binaries in C, 2.6x faster again A kernel cannot build terms, but it can make the input easy for the BIFs: a cleaned copy with byte 31 between fields and byte 30 after each row, @@ -2917,10 +2917,10 @@ costs as much as the splits. - **Numbers** (20 MB, a 200k-row export with quoted fields; a fresh process per parse): NimbleCSV 308-315 ms; the best Elixir (lines by `:binary.split`, unquoted lines split on commas, quoted lines walked with - `:binary.match`) 288-299 ms; the cleaned copy 131-142 ms end to end - (2.23x; the copy 36 ms, the splits 125-131 ms); an `[i64]` tape of start + `:binary.match`) 274-299 ms; the cleaned copy 131-142 ms end to end + (2.09-2.23x; the copy 36 ms, the splits 125-131 ms); an `[i64]` tape of start and length per field 152-158 ms (scan 28 ms, rows from it 119-127 ms; - only walking its 1.4M entries took 39 ms); rusty_csv 0.4.6 49.6-53.5 ms. + only walking its 1.4M entries took 39 ms); rusty_csv 0.4.6 49.5-53.5 ms. - **Heap state decides term-building benchmarks:** timed in one long-lived process, best of 15, the same code gave NimbleCSV 134-358 ms and the copy 96-264 ms, 0.81x to 2.08x between runs, with or without a GC before each @@ -2931,10 +2931,10 @@ costs as much as the splits. sub-binaries cost little. For CSV to rows, rusty_csv is the fastest; a kernel's cleaned copy is a middle option without another NIF, and yields on a normal scheduler. JSON to terms is not measured. -- **Conditions:** M-series Mac, OTP 29, Elixir 1.20.2; three runs with a +- **Conditions:** M-series Mac, OTP 29, Elixir 1.20.2; four runs with a fresh process, five before in a long-lived one. - **Source:** [`tape_csv.exs`](https://github.com/DanielLuning/defk/blob/main/examples/tape_csv.exs), `defk-9p6.30`. -- **Status:** provisional until the stored run. +- **Status:** measured (the stored run, quiet); the rest provisional. --- diff --git a/examples/README.md b/examples/README.md index 38ebd05a..acd4e737 100644 --- a/examples/README.md +++ b/examples/README.md @@ -43,7 +43,7 @@ script tried and what to take from it; the numbers behind them are in each script's header. -Measured on Apple M4 Pro (12 logical cores) in 3 runs, from 2026-10-19 to 2026-10-20. The newest at commit `0d6e7c78`, OTP 29.0.3, Elixir 1.20.2: `mix defk.examples --bench --only numeric_loader --snapshot --fresh`. A quiet measurement: other processes used at most 1.0 cores before and after it. A row from an earlier run says when it was measured, and at which commit. +Measured on Apple M4 Pro (12 logical cores) in 4 runs, from 2026-10-19 to 2026-10-20. The newest at commit `71e29777`, OTP 29.0.3, Elixir 1.20.2: `mix defk.examples --bench --only tape_csv --snapshot --fresh`. A quiet measurement: other processes used at most 1.0 cores before and after it. A row from an earlier run says when it was measured, and at which commit. | | Example | The job | defk vs best Elixir | vs the package's NIF | In one line | |---|---|---|---|---|---| @@ -67,9 +67,10 @@ Measured on Apple M4 Pro (12 logical cores) in 3 runs, from 2026-10-19 to 2026-1 | ✅ | [`monte_carlo_rtp`](monte_carlo_rtp.exs) (measured 2026-10-19 at `684a3592`) | 100M slot spins on all schedulers | **3.9×** | – | ≈1× at 1M spins: short kernel chunks stay on few schedulers | | ✅ | [`per_pair_distance`](per_pair_distance.exs) (measured 2026-10-19 at `8fa900b7`) | Levenshtein on 50k short pairs, one call each | **3.6×** | – | stay in Elixir for very short strings | | ✅ | [`geo_nearest`](geo_nearest.exs) (measured 2026-10-19 at `8fa900b7`) | 10 nearest of 500k locations | **3.2×** | – | 14× in one process; a grid prefilter helps both sides | -| ✅ | [`numeric_loader`](numeric_loader.exs) | numeric CSV of 26 MB into packed f64s | **2.7×** | – | a JSON array of numbers 4.0×; a fixed number grammar, and NimbleCSV or Jason for anything else | +| ✅ | [`numeric_loader`](numeric_loader.exs) (measured 2026-10-20 at `0d6e7c78`) | numeric CSV of 26 MB into packed f64s | **2.7×** | – | a JSON array of numbers 4.0×; a fixed number grammar, and NimbleCSV or Jason for anything else | | ✅ | [`cdcl_sat_per_call`](cdcl_sat_per_call.exs) (measured 2026-10-19 at `8fa900b7`) | one SAT solve per call, 26–28 vars | **2.6×** | 1.6× picosat | ≈1× on tiny formulas; batch the solves when one question makes several | | ✅ | [`gorilla_encode`](gorilla_encode.exs) (measured 2026-10-19 at `684a3592`) | Gorilla encoding of 100-point blocks | **2.4×** | 4.2× gorilla_stream's NIF | close to the bar at this size; 4.1× at 1M points | +| ✅ | [`tape_csv`](tape_csv.exs) | CSV of 20 MB to NimbleCSV's rows, a fresh process per parse | **2.1×** | 0.38× rusty_csv | in a long-lived process 0.8–2.1× run to run; an [i64] tape of field events is slower than the cleaned copy | | ✅ | [`fuzzy_search/`](fuzzy_search) | a search service over 234k words behind Bandit | its own measurement, see its README | – | Myers on both sides; `/health` latency stays at Elixir's while a blocking NIF raises it | | 🟡 | [`batched_levenshtein`](batched_levenshtein.exs) (measured 2026-10-19 at `8fa900b7`) | one query against 234k words, 12 chunks each side | 1.5× | 13× ex_strsim per pair | Myers bit-parallel in plain Elixir is close; use it | | 🟡 | [`ws_masking`](ws_masking.exs) (measured 2026-10-19 at `8fa900b7`) | mask 125-byte WebSocket frames | 1.5× | 1.2× bandit_native | the call is the cost; 3.4× at 1 MiB, crossover 1024 B | @@ -196,7 +197,7 @@ has them in full. - [`tape_csv.exs`](tape_csv.exs): CSV to NimbleCSV's rows of binaries through a kernel tape. An `[i64]` tape of field events loses to a cleaned copy (separators as bytes 30 and 31) that two `:binary.split` - passes turn into rows. In a fresh process per parse the copy is 2.2× the + passes turn into rows. In a fresh process per parse the copy is 2.1–2.2× the best Elixir; rusty_csv's NIF builds the same sub-binaries in C, 2.6× faster again. Timed in one long-lived process, the verdict moved from 0.8× to 2.1× run to run with the heap the earlier runs left. @@ -305,7 +306,7 @@ Nothing here is excluded on principle: each row names the condition that would r | Job | Wrapped today | Why not from scratch | What would change it | |---|---|---|---| | Signing, key exchange, secret-dependent crypto (secp256k1, ed25519, X25519, AEAD) | libsodium, secp256k1, ring via Rustler; `:crypto` | The one rule says nothing about timing: clang `-O3` may turn a select into a branch or a table lookup into a secret-dependent access, and a process dump or yielded frame holds key material. Public-input verification is allowed but gains little. | A constant-time mode: a restricted subset checked at compile time, codegen verified with dudect or ctgrind in CI, frame zeroisation, and an external audit. Until then, `:crypto`. | -| Parsers that produce trees or term streams (HTML, Markdown, tree-sitter, JSON/CSV to terms, protobuf, msgpack) | html5ever, comrak, tree-sitter, jsont, rusty_csv, protox | The product is BEAM terms and building them is the cost: `docs/research/string-workloads.md` measured a 13x gap between scanning and term creation. [`tape_csv`](tape_csv.exs) measured the tape route for CSV: a kernel cleans 20 MB in 36 ms, and two `:binary.split` passes then take 125–131 ms to build the rows, 2.2× the best Elixir in a fresh process per parse (in one long-lived process 0.8–2.1×, with the heap). rusty_csv builds the same sub-binaries in C, 2.6× faster again: a NIF builds terms far cheaper than Elixir does. A kernel cannot build terms. | Reopened narrowly for CSV to rows (`tape_csv`, good) where an extra NIF is not wanted; rusty_csv stays the fastest. JSON to terms is not measured. BPE tokenisation to `[i64]` ids is the exception: its product is packed, and so is a numeric CSV or JSON number array loaded into f64s (`numeric_loader`, good: about 2.7× and 4×). | +| Parsers that produce trees or term streams (HTML, Markdown, tree-sitter, JSON/CSV to terms, protobuf, msgpack) | html5ever, comrak, tree-sitter, jsont, rusty_csv, protox | The product is BEAM terms and building them is the cost: `docs/research/string-workloads.md` measured a 13x gap between scanning and term creation. [`tape_csv`](tape_csv.exs) measured the tape route for CSV: a kernel cleans 20 MB in 36 ms, and two `:binary.split` passes then take 125–131 ms to build the rows, 2.1–2.2× the best Elixir in a fresh process per parse (in one long-lived process 0.8–2.1×, with the heap). rusty_csv builds the same sub-binaries in C, 2.6× faster again: a NIF builds terms far cheaper than Elixir does. A kernel cannot build terms. | Reopened narrowly for CSV to rows (`tape_csv`, good) where an extra NIF is not wanted; rusty_csv stays the fastest. JSON to terms is not measured. BPE tokenisation to `[i64]` ids is the exception: its product is packed, and so is a numeric CSV or JSON number array loaded into f64s (`numeric_loader`, good: about 2.7× and 4×). | | Regular expressions | `:re` (PCRE in C), Rust regex via Rustler | `:re` is already native and compiled once per pattern; a kernel cannot take a pattern as data without becoming an interpreter, and fixed-string search is `:binary.match`. | A specific pattern class with a known kernel form (glob, or a simple byte DFA compiled at Elixir compile time) and a benchmark showing `:re` loses on it. | | Large codecs and encoders (brotli, zstd encode, audio and video, PDF/typst rendering) | brotli, zstd, membrane_* FFmpeg/opus bindings, typst | The cost is spec surface and tuning, not the loop: 10^4-10^5 lines with SIMD everywhere. A kernel would be a worse implementation of a solved problem. | A narrow, named need: one compression level with fixed tables (zstd level 1 is the lz4-encode ticket plus FSE) or a decoder-only requirement for untrusted input. Each reopens one row, not the category. | | Systems: databases, dataframes, tensor compilers (SQLite, DuckDB, Polars/Explorer, XLA/EXLA, LibTorch) | exqlite, duckdbex, explorer, exla, torchx | The value is the library: query planners, columnar engines, autodiff, device backends. defk is a kernel, not a system. | Interop, not rewriting: zero-copy views over their packed buffers so a per-element loop beside them runs as a kernel. `docs/research/nx-boundary.md` says where that line is; file an interop ticket when a workload asks. | diff --git a/examples/scoreboard.json b/examples/scoreboard.json index 3a4872ad..e222dec5 100644 --- a/examples/scoreboard.json +++ b/examples/scoreboard.json @@ -33,6 +33,7 @@ {"name":"poor_tiny_calls","quiet":true,"results":[{"baseline":"optimized","cache":"stored","correct":true,"defk_ms":53.406,"defk_spread_ms":{"max":87.449,"median":80.758,"runs":5},"elixir_ms":14.797,"elixir_spread_ms":{"max":22.138,"median":15.554,"runs":5},"example":"poor_tiny_calls","expected":"poor","reason":"one NIF call per element: the call overhead is larger than the work","speedup":0.28,"stamp":{"c":{"Clamp.Kernels":"ccc3a59eb65b3080b5b00482aca8813c"},"commit":"8fa900b7","defk":"0.1.0","measured_at":"2026-10-19T11:40:42Z","quiet":{"limit":1.0,"other_cpu":[0.06,0.07],"quiet":true,"waited_s":0}},"verdict":"poor"}],"run":{"command":"mix defk.examples --bench --all --snapshot","commit":"8fa900b7","elixir":"1.20.2","measured_at":"2026-10-19T11:41:41Z","otp":"29.0.3","quiet_limit":1.0},"status":"ok"}, {"name":"rolling_stats","quiet":true,"results":[{"baseline":"optimized","cache":"stored","correct":true,"defk_ms":1.089,"defk_spread_ms":{"max":1.88,"median":1.701,"runs":5},"elixir_ms":10.252,"elixir_spread_ms":{"max":11.131,"median":10.366,"runs":5},"example":"rolling_stats","expected":"good","reason":"one pass over packed floats, no allocation per sample","speedup":9.41,"stamp":{"c":{"Rolling.Kernels":"741787c6de035e89fde5d5b14392c9f2"},"commit":"8fa900b7","defk":"0.1.0","measured_at":"2026-10-19T11:40:43Z","quiet":{"limit":1.0,"other_cpu":[0.34,0.07],"quiet":true,"waited_s":0}},"verdict":"good"}],"run":{"command":"mix defk.examples --bench --all --snapshot","commit":"8fa900b7","elixir":"1.20.2","measured_at":"2026-10-19T11:41:41Z","otp":"29.0.3","quiet_limit":1.0},"status":"ok"}, {"name":"sort_u64_keys","quiet":true,"results":[{"baseline":"optimized","cache":"stored","correct":true,"defk_ms":59.478,"defk_spread_ms":{"max":60.181,"median":60.172,"runs":5},"elixir_ms":355.392,"elixir_spread_ms":{"max":418.879,"median":369.938,"runs":5},"example":"sort_packed","expected":"good","reason":"packed in, packed out: no list is ever built on the defk side","speedup":5.98,"stamp":{"c":{"Keys.Kernels":"a49c2e57190d29d9d5d46c60ecd6535f"},"commit":"684a3592","defk":"0.1.0","measured_at":"2026-10-19T11:59:29Z","quiet":{"limit":1.0,"other_cpu":[0.12,0.04],"quiet":true,"waited_s":0}},"verdict":"good"},{"baseline":"optimized","cache":"reused","correct":true,"defk_ms":94.191,"defk_spread_ms":{"max":118.427,"median":96.442,"runs":5},"elixir_ms":362.119,"elixir_spread_ms":{"max":396.991,"median":371.298,"runs":5},"example":"sort_to_list","expected":"good","measured_at":"2026-10-19T11:41:02Z","reason":"decoding the sorted result to a list costs 40 ms, the margin shrinks to about 2.5x but holds","speedup":3.84,"stamp":{"c":{"Keys.Kernels":"a49c2e57190d29d9d5d46c60ecd6535f"},"commit":"8fa900b7","defk":"0.1.0","measured_at":"2026-10-19T11:41:02Z","quiet":{"limit":1.0,"other_cpu":[0.54,0.22],"quiet":true,"waited_s":6}},"verdict":"good"}],"run":{"command":"mix defk.examples --bench --only codecs,gorilla_encode,monte_carlo_rtp,sort_u64_keys --snapshot","commit":"684a3592","elixir":"1.20.2","measured_at":"2026-10-19T11:59:29Z","otp":"29.0.3","quiet_limit":1.0},"status":"ok"}, + {"name":"tape_csv","quiet":true,"results":[{"baseline":"optimized","cache":"stored","correct":true,"crossover":"1000000 rows","defk_ms":131.174,"defk_spread_ms":{"max":133.365,"median":132.849,"runs":15},"elixir_ms":274.473,"elixir_spread_ms":{"max":279.982,"median":277.864,"runs":15},"example":"tape_csv","expected":"good","native_label":"rusty_csv 0.4.6","native_ms":49.459,"native_speedup":0.38,"native_spread_ms":{"max":52.635,"median":50.216,"runs":15},"realistic_size":"20 MB: a 200k-row customer export","reason":"the kernel cleans 20 MB in 36 ms and two :binary.split passes build the rows, with less garbage than a parser in Elixir; rusty_csv builds them in C, 2.7x faster still","speedup":2.09,"stamp":{"c":{"Tape.Kernels":"9a6a5466f9da258adf6f944d858351d0"},"commit":"71e29777","defk":"0.1.0","measured_at":"2026-10-20T13:41:22Z","quiet":{"limit":1.0,"other_cpu":[0.29,0.82],"quiet":true,"waited_s":0}},"sweep":[{"defk_ms":1.082,"elixir_ms":1.71,"size":"2000 rows","speedup":1.58},{"defk_ms":10.814,"elixir_ms":17.611,"size":"20000 rows","speedup":1.63},{"defk_ms":646.619,"elixir_ms":1524.704,"size":"1000000 rows","speedup":2.36}],"verdict":"good"}],"run":{"command":"mix defk.examples --bench --only tape_csv --snapshot --fresh","commit":"71e29777","elixir":"1.20.2","measured_at":"2026-10-20T13:41:22Z","otp":"29.0.3","quiet_limit":1.0},"status":"ok"}, {"name":"utf8_valid","quiet":true,"results":[{"baseline":"optimized","cache":"stored","correct":true,"crossover":null,"defk_ms":26.992,"defk_spread_ms":{"max":27.922,"median":27.5,"runs":15},"elixir_ms":28.037,"elixir_spread_ms":{"max":29.813,"median":29.156,"runs":15},"example":"utf8_valid","expected":"marginal","realistic_size":"16 MB","reason":"a scalar byte DFA competes with a C BIF; :unicode.characters_to_binary is already C and the DFA is only about 1.07x ahead, so use String.valid? (a SIMD validator is a wrapper job)","speedup":1.04,"stamp":{"c":{"Utf8.Kernels":"5439915fa11707ea174a3f68e6e8af0b"},"commit":"8fa900b7","defk":"0.1.0","measured_at":"2026-10-19T11:41:05Z","quiet":{"limit":1.0,"other_cpu":[0.14,0.38],"quiet":true,"waited_s":0}},"sweep":[{"defk_ms":0.002,"elixir_ms":0.0,"size":1024,"speedup":0.0},{"defk_ms":0.102,"elixir_ms":0.057,"size":65536,"speedup":0.56},{"defk_ms":1.718,"elixir_ms":1.812,"size":1048576,"speedup":1.05},{"defk_ms":27.234,"elixir_ms":29.351,"size":16777216,"speedup":1.08}],"verdict":"marginal"}],"run":{"command":"mix defk.examples --bench --all --snapshot","commit":"8fa900b7","elixir":"1.20.2","measured_at":"2026-10-19T11:41:41Z","otp":"29.0.3","quiet_limit":1.0},"status":"ok"}, {"name":"ws_masking","quiet":true,"results":[{"baseline":"optimized","cache":"stored","correct":true,"crossover":"1024 B","defk_gap_ms":{"max":1.751,"p99":1.751},"defk_ms":1.231,"defk_spread_ms":{"max":1.369,"median":1.287,"runs":15},"elixir_gap_ms":{"max":2.008,"p99":2.008},"elixir_ms":1.793,"elixir_spread_ms":{"max":2.786,"median":2.397,"runs":15},"example":"ws_masking","expected":"marginal","native_gap_ms":{"max":8.345,"p99":8.345},"native_label":"bandit_native 0.1.2 Bandit.Native.mask/2 (Rust, byte at a time, normal scheduler)","native_ms":1.424,"native_speedup":1.16,"native_spread_ms":{"max":1.556,"median":1.461,"runs":15},"realistic_size":"125 B frames (the largest 7-bit length), 10,000 frames per timed run","reason":"a 125 B frame is ~0.13 us of kernel call against the BEAM's already-compiled 32-bit loop; use Bandit's own ws_mask/2 for small frames","speedup":1.46,"stamp":{"c":{"Ws.Defk":"aa97b7005e38e3435e015619bd5af1e8"},"commit":"8fa900b7","defk":"0.1.0","measured_at":"2026-10-19T11:41:21Z","quiet":{"limit":1.0,"other_cpu":[0.34,0.11],"quiet":true,"waited_s":8}},"sweep":[{"defk_ms":1.226,"elixir_ms":1.626,"size":"125 B","speedup":1.33},{"defk_ms":2.176,"elixir_ms":5.996,"size":"1024 B","speedup":2.76},{"defk_ms":1.35,"elixir_ms":4.582,"size":"8192 B","speedup":3.39},{"defk_ms":1.222,"elixir_ms":3.95,"size":"65536 B","speedup":3.23},{"defk_ms":1.203,"elixir_ms":4.109,"size":"1048576 B","speedup":3.42},{"defk_ms":2.547,"elixir_ms":9.134,"size":"16777216 B","speedup":3.59}],"verdict":"marginal"},{"baseline":"optimized","cache":"stored","correct":true,"defk_ms":1.2,"defk_spread_ms":{"max":1.23,"median":1.209,"runs":15},"elixir_ms":4.103,"elixir_spread_ms":{"max":7.095,"median":4.194,"runs":15},"example":"ws_masking_1mib","expected":"good","native_label":"bandit_native 0.1.2 Bandit.Native.mask/2 (Rust, byte at a time, normal scheduler)","native_ms":4.215,"native_speedup":3.51,"native_spread_ms":{"max":4.418,"median":4.38,"runs":15},"realistic_size":"1 MiB frames, 8 per timed run","reason":"the vectorised kernel runs at memory bandwidth (~38 GB/s) while the best Elixir (:crypto.exor/2) and bandit_native's byte-at-a-time loop run at 2 GB/s","speedup":3.42,"stamp":{"c":{"Ws.Defk":"aa97b7005e38e3435e015619bd5af1e8"},"commit":"8fa900b7","defk":"0.1.0","measured_at":"2026-10-19T11:41:21Z","quiet":{"limit":1.0,"other_cpu":[0.11,0.06],"quiet":true,"waited_s":0}},"verdict":"good"}],"run":{"command":"mix defk.examples --bench --all --snapshot","commit":"8fa900b7","elixir":"1.20.2","measured_at":"2026-10-19T11:41:41Z","otp":"29.0.3","quiet_limit":1.0},"status":"ok"} ] diff --git a/examples/tape_csv.exs b/examples/tape_csv.exs index 2ce5375c..0deba2b7 100644 --- a/examples/tape_csv.exs +++ b/examples/tape_csv.exs @@ -56,20 +56,20 @@ # and the spread is 2% (median against best). # # Measured (M-series Mac, OTP 29, Elixir 1.20, 2026-10-20, fresh process -# per run; the report best of 15, the variant lines best of 5; three runs -# with E2, provisional until the stored run): +# per run; the report best of 15, the variant lines best of 5; four runs +# with E2, the last stored and quiet; the others provisional): # -# 20 MB E1 308-315 ms, E2 288-299 ms, E3 300-305 ms; -# the copy 130.7-141.9 ms 2.23x over E2 good -# rusty_csv 49.6-53.5 ms: 5.8x E2, 2.6x the copy. +# 20 MB E1 308-315 ms, E2 274-299 ms, E3 300-305 ms; +# the copy 130.7-141.9 ms 2.09-2.23x over E2 good +# rusty_csv 49.5-53.5 ms: 5.5-5.9x E2, 2.6-2.7x the copy. # The parts pass 1 4.8-5.1 ms, scan to the [i64] tape 28 ms, rows # (best of 5) from it 119-127 ms (the tape 152-158 ms end to end); # the copy 36 ms, its two splits 125-131 ms. -# Sweep 2k rows 1.7x, 20k 1.7x, 1M rows (100 MB) 2.5x. +# Sweep 2k rows 1.6-1.7x, 20k 1.6-1.7x, 1M rows (100 MB) 2.4-2.5x. # # What it means: a kernel scans CSV at 0.5-0.7 GB/s, and Elixir then # spends 3.5-4.5x the scan building the rows. The cleaned copy, split by -# the BIFs, beats the best Elixir by about 2.2x in a fresh process, so a +# the BIFs, beats the best Elixir by about 2.1-2.2x in a fresh process, so a # tape does reopen the wrapper table's row, narrowly. rusty_csv builds the # same sub-binaries in C, 2.6x faster again, on a dirty scheduler: where # the speed matters, it is still the choice. The kernel route needs no