diff --git a/README.md b/README.md index b30c64d1..6c9644ef 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,7 @@ Speedups over an optimised Elixir baseline, measured end to end on Apple M4 Pro | ✅ | base64, hex and base58 codecs (4.2× the fast64 NIF); Snappy and LZ4 block decode on untrusted input | 5.1–6.1× | | ✅ | sorting packed keys, nearest of 500k locations, Monte Carlo slot simulation with `:rand` bit for bit, Gorilla encoding | 2.4–6.0× | | ✅ | a numeric CSV or a JSON array of numbers straight into packed f64s (NimbleCSV and Jason stay for anything else) | 2.7× | +| ✅ | CSV to NimbleCSV's rows through a kernel that cleans the separators for `:binary.split` (rusty_csv's NIF is faster still) | 2.1× | | ✅ | a fuzzy-search web service, Myers bit-parallel on both sides: `/health` latency stays at Elixir's while a blocking NIF raises it | [its own measurement](https://github.com/DanielLuning/defk/blob/main/examples/fuzzy_search/README.md) | | 🟡 | a dictionary-wide Levenshtein batch over all schedulers: Myers bit-parallel in Elixir is close | 1.5× | | 🟡 | small WebSocket frames, UTF-8 validation, substring search: the BIFs are native already | 1.0–1.5× | diff --git a/docs/performance.md b/docs/performance.md index ee530d5b..56eca450 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -2917,6 +2917,36 @@ output array would). - **Source:** [`bench/port_roundtrip.exs`](https://github.com/DanielLuning/defk/blob/main/bench/port_roundtrip.exs), [`bench/port_echo.c`](https://github.com/DanielLuning/defk/blob/main/bench/port_echo.c), `defk-7dm`. - **Status:** one quiet run; rerun with `defk-sgw.4`'s snapshot if the verdict cites it. +### 7.11 CSV to terms: a kernel that cleans the separators for `:binary.split` is 2.1-2.2x the best Elixir; rusty_csv builds the sub-binaries in C, 2.6x faster again + +A kernel cannot build terms, but it can make the input easy for the BIFs: +a cleaned copy with byte 31 between fields and byte 30 after each row, +quotes removed, that two `:binary.split/3` passes turn into NimbleCSV's +rows. A tape of field events walked in Elixir is slower: reading the tape +costs as much as the splits. + +- **Numbers** (20 MB, a 200k-row export with quoted fields; a fresh process + per parse): NimbleCSV 308-315 ms; the best Elixir (lines by + `:binary.split`, unquoted lines split on commas, quoted lines walked with + `:binary.match`) 274-299 ms; the cleaned copy 131-142 ms end to end + (2.09-2.23x; the copy 36 ms, the splits 125-131 ms); an `[i64]` tape of start + and length per field 152-158 ms (scan 28 ms, rows from it 119-127 ms; + only walking its 1.4M entries took 39 ms); rusty_csv 0.4.6 49.5-53.5 ms. +- **Heap state decides term-building benchmarks:** timed in one long-lived + process, best of 15, the same code gave NimbleCSV 134-358 ms and the copy + 96-264 ms, 0.81x to 2.08x between runs, with or without a GC before each + run. A fresh process per run grows its heap from nothing every time; the + spread was 2% (median against best). +- **What it means:** the 13x gap between scanning and term creation that + `string-workloads.md` measured holds for Elixir, not for C: rusty_csv's + sub-binaries cost little. For CSV to rows, rusty_csv is the fastest; a + kernel's cleaned copy is a middle option without another NIF, and yields + on a normal scheduler. JSON to terms is not measured. +- **Conditions:** M-series Mac, OTP 29, Elixir 1.20.2; four runs with a + fresh process, five before in a long-lived one. +- **Source:** [`tape_csv.exs`](https://github.com/DanielLuning/defk/blob/main/examples/tape_csv.exs), `defk-9p6.30`. +- **Status:** measured (the stored run, quiet); the rest provisional. + --- ## 8. Myths we disproved and open questions diff --git a/examples/README.md b/examples/README.md index 26089812..7f88ee52 100644 --- a/examples/README.md +++ b/examples/README.md @@ -43,7 +43,7 @@ script tried and what to take from it; the numbers behind them are in each script's header. -Measured on Apple M4 Pro (12 logical cores) in 3 runs, from 2026-10-19 to 2026-10-20. The newest at commit `0d6e7c78`, OTP 29.0.3, Elixir 1.20.2: `mix defk.examples --bench --only numeric_loader --snapshot --fresh`. A quiet measurement: other processes used at most 1.0 cores before and after it. A row from an earlier run says when it was measured, and at which commit. +Measured on Apple M4 Pro (12 logical cores) in 4 runs, from 2026-10-19 to 2026-10-20. The newest at commit `71e29777`, OTP 29.0.3, Elixir 1.20.2: `mix defk.examples --bench --only tape_csv --snapshot --fresh`. A quiet measurement: other processes used at most 1.0 cores before and after it. A row from an earlier run says when it was measured, and at which commit. | | Example | The job | defk vs best Elixir | vs the package's NIF | In one line | |---|---|---|---|---|---| @@ -67,9 +67,10 @@ Measured on Apple M4 Pro (12 logical cores) in 3 runs, from 2026-10-19 to 2026-1 | ✅ | [`monte_carlo_rtp`](monte_carlo_rtp.exs) (measured 2026-10-19 at `684a3592`) | 100M slot spins on all schedulers | **3.9×** | – | ≈1× at 1M spins: short kernel chunks stay on few schedulers | | ✅ | [`per_pair_distance`](per_pair_distance.exs) (measured 2026-10-19 at `8fa900b7`) | Levenshtein on 50k short pairs, one call each | **3.6×** | – | stay in Elixir for very short strings | | ✅ | [`geo_nearest`](geo_nearest.exs) (measured 2026-10-19 at `8fa900b7`) | 10 nearest of 500k locations | **3.2×** | – | 14× in one process; a grid prefilter helps both sides | -| ✅ | [`numeric_loader`](numeric_loader.exs) | numeric CSV of 26 MB into packed f64s | **2.7×** | – | a JSON array of numbers 4.0×; a fixed number grammar, and NimbleCSV or Jason for anything else | +| ✅ | [`numeric_loader`](numeric_loader.exs) (measured 2026-10-20 at `0d6e7c78`) | numeric CSV of 26 MB into packed f64s | **2.7×** | – | a JSON array of numbers 4.0×; a fixed number grammar, and NimbleCSV or Jason for anything else | | ✅ | [`cdcl_sat_per_call`](cdcl_sat_per_call.exs) (measured 2026-10-19 at `8fa900b7`) | one SAT solve per call, 26–28 vars | **2.6×** | 1.6× picosat | ≈1× on tiny formulas; batch the solves when one question makes several | | ✅ | [`gorilla_encode`](gorilla_encode.exs) (measured 2026-10-19 at `684a3592`) | Gorilla encoding of 100-point blocks | **2.4×** | 4.2× gorilla_stream's NIF | close to the bar at this size; 4.1× at 1M points | +| ✅ | [`tape_csv`](tape_csv.exs) | CSV of 20 MB to NimbleCSV's rows, a fresh process per parse | **2.1×** | 0.38× rusty_csv | in a long-lived process 0.8–2.1× run to run; an [i64] tape of field events is slower than the cleaned copy | | ✅ | [`fuzzy_search/`](fuzzy_search) | a search service over 234k words behind Bandit | its own measurement, see its README | – | Myers on both sides; `/health` latency stays at Elixir's while a blocking NIF raises it | | 🟡 | [`batched_levenshtein`](batched_levenshtein.exs) (measured 2026-10-19 at `8fa900b7`) | one query against 234k words, 12 chunks each side | 1.5× | 13× ex_strsim per pair | Myers bit-parallel in plain Elixir is close; use it | | 🟡 | [`ws_masking`](ws_masking.exs) (measured 2026-10-19 at `8fa900b7`) | mask 125-byte WebSocket frames | 1.5× | 1.2× bandit_native | the call is the cost; 3.4× at 1 MiB, crossover 1024 B | @@ -193,6 +194,13 @@ has them in full. 5.7–5.9× faster than NimbleCSV with `String.to_float`; the kernel is 2.7× that on CSV, and about 4× on JSON against Jason, OTP's `:json` and the same parser. +- [`tape_csv.exs`](tape_csv.exs): CSV to NimbleCSV's rows of binaries + through a kernel tape. An `[i64]` tape of field events loses to a + cleaned copy (separators as bytes 30 and 31) that two `:binary.split` + passes turn into rows. In a fresh process per parse the copy is 2.1–2.2× the + best Elixir; rusty_csv's NIF builds the same sub-binaries in C, 2.6× + faster again. Timed in one long-lived process, the verdict moved from + 0.8× to 2.1× run to run with the heap the earlier runs left. - [`lz4_snappy_decode.exs`](lz4_snappy_decode.exs): Snappy and LZ4 block decode from scratch on untrusted input, against snappyer, lz4_erl and nimble_lz4. Against the NIFs as installed from Hex (built with no `-O`) @@ -298,7 +306,7 @@ Nothing here is excluded on principle: each row names the condition that would r | Job | Wrapped today | Why not from scratch | What would change it | |---|---|---|---| | Signing, key exchange, secret-dependent crypto (secp256k1, ed25519, X25519, AEAD) | libsodium, secp256k1, ring via Rustler; `:crypto` | The one rule says nothing about timing: clang `-O3` may turn a select into a branch or a table lookup into a secret-dependent access, and a process dump or yielded frame holds key material. Public-input verification is allowed but gains little. | A constant-time mode: a restricted subset checked at compile time, codegen verified with dudect or ctgrind in CI, frame zeroisation, and an external audit. Until then, `:crypto`. | -| Parsers that produce trees or term streams (HTML, Markdown, tree-sitter, JSON/CSV to terms, protobuf, msgpack) | html5ever, comrak, tree-sitter, jsont, rusty_csv, protox | The product is BEAM terms and building them is the cost: `docs/research/string-workloads.md` measured a 13x gap between scanning and term creation, and a NIF builds terms serially at the same price. A kernel cannot build terms, and if it could it would be no faster. | A "tape" output (flat `[i64]` of kind/start/len events, of which a fused CSV parser is one form), plus a benchmark showing tape plus Elixir tree building beats Jason or NimbleCSV end to end. BPE tokenisation to `[i64]` ids is the exception: its product is packed, and so is a numeric CSV or JSON number array loaded into f64s (`numeric_loader`, good: about 2.7× and 4×). | +| Parsers that produce trees or term streams (HTML, Markdown, tree-sitter, JSON/CSV to terms, protobuf, msgpack) | html5ever, comrak, tree-sitter, jsont, rusty_csv, protox | The product is BEAM terms and building them is the cost: `docs/research/string-workloads.md` measured a 13x gap between scanning and term creation. [`tape_csv`](tape_csv.exs) measured the tape route for CSV: a kernel cleans 20 MB in 36 ms, and two `:binary.split` passes then take 125–131 ms to build the rows, 2.1–2.2× the best Elixir in a fresh process per parse (in one long-lived process 0.8–2.1×, with the heap). rusty_csv builds the same sub-binaries in C, 2.6× faster again: a NIF builds terms far cheaper than Elixir does. A kernel cannot build terms. | Reopened narrowly for CSV to rows (`tape_csv`, good) where an extra NIF is not wanted; rusty_csv stays the fastest. JSON to terms is not measured. BPE tokenisation to `[i64]` ids is the exception: its product is packed, and so is a numeric CSV or JSON number array loaded into f64s (`numeric_loader`, good: about 2.7× and 4×). | | Regular expressions | `:re` (PCRE in C), Rust regex via Rustler | `:re` is already native and compiled once per pattern; a kernel cannot take a pattern as data without becoming an interpreter, and fixed-string search is `:binary.match`. | A specific pattern class with a known kernel form (glob, or a simple byte DFA compiled at Elixir compile time) and a benchmark showing `:re` loses on it. | | Large codecs and encoders (brotli, zstd encode, audio and video, PDF/typst rendering) | brotli, zstd, membrane_* FFmpeg/opus bindings, typst | The cost is spec surface and tuning, not the loop: 10^4-10^5 lines with SIMD everywhere. A kernel would be a worse implementation of a solved problem. | A narrow, named need: one compression level with fixed tables (zstd level 1 is the lz4-encode ticket plus FSE) or a decoder-only requirement for untrusted input. Each reopens one row, not the category. | | Systems: databases, dataframes, tensor compilers (SQLite, DuckDB, Polars/Explorer, XLA/EXLA, LibTorch) | exqlite, duckdbex, explorer, exla, torchx | The value is the library: query planners, columnar engines, autodiff, device backends. defk is a kernel, not a system. | Interop, not rewriting: zero-copy views over their packed buffers so a per-element loop beside them runs as a kernel. `docs/research/nx-boundary.md` says where that line is; file an interop ticket when a workload asks. | diff --git a/examples/_tables.exs b/examples/_tables.exs index 375e74a5..bcd52d89 100644 --- a/examples/_tables.exs +++ b/examples/_tables.exs @@ -160,6 +160,11 @@ job: "sort 1M packed keys", line: "{speedup:sort_to_list} when the caller needs a list back" }, + "tape_csv" => %{ + job: "CSV of 20 MB to NimbleCSV's rows, a fresh process per parse", + native: "{native} rusty_csv", + line: "in a long-lived process 0.8–2.1× run to run; an [i64] tape of field events is slower than the cleaned copy" + }, "utf8_valid" => %{ job: "validate 16 MB of UTF-8", line: "`String.valid?` is C already" @@ -227,6 +232,11 @@ "a numeric CSV or a JSON array of numbers straight into packed f64s (NimbleCSV and Jason stay for anything else)", examples: ["numeric_loader"] }, + %{ + text: + "CSV to NimbleCSV's rows through a kernel that cleans the separators for `:binary.split` (rusty_csv's NIF is faster still)", + examples: ["tape_csv"] + }, %{ text: "a fuzzy-search web service, Myers bit-parallel on both sides: `/health` latency stays at Elixir's while a blocking NIF raises it", diff --git a/examples/scoreboard.json b/examples/scoreboard.json index 3a4872ad..e222dec5 100644 --- a/examples/scoreboard.json +++ b/examples/scoreboard.json @@ -33,6 +33,7 @@ {"name":"poor_tiny_calls","quiet":true,"results":[{"baseline":"optimized","cache":"stored","correct":true,"defk_ms":53.406,"defk_spread_ms":{"max":87.449,"median":80.758,"runs":5},"elixir_ms":14.797,"elixir_spread_ms":{"max":22.138,"median":15.554,"runs":5},"example":"poor_tiny_calls","expected":"poor","reason":"one NIF call per element: the call overhead is larger than the work","speedup":0.28,"stamp":{"c":{"Clamp.Kernels":"ccc3a59eb65b3080b5b00482aca8813c"},"commit":"8fa900b7","defk":"0.1.0","measured_at":"2026-10-19T11:40:42Z","quiet":{"limit":1.0,"other_cpu":[0.06,0.07],"quiet":true,"waited_s":0}},"verdict":"poor"}],"run":{"command":"mix defk.examples --bench --all --snapshot","commit":"8fa900b7","elixir":"1.20.2","measured_at":"2026-10-19T11:41:41Z","otp":"29.0.3","quiet_limit":1.0},"status":"ok"}, {"name":"rolling_stats","quiet":true,"results":[{"baseline":"optimized","cache":"stored","correct":true,"defk_ms":1.089,"defk_spread_ms":{"max":1.88,"median":1.701,"runs":5},"elixir_ms":10.252,"elixir_spread_ms":{"max":11.131,"median":10.366,"runs":5},"example":"rolling_stats","expected":"good","reason":"one pass over packed floats, no allocation per sample","speedup":9.41,"stamp":{"c":{"Rolling.Kernels":"741787c6de035e89fde5d5b14392c9f2"},"commit":"8fa900b7","defk":"0.1.0","measured_at":"2026-10-19T11:40:43Z","quiet":{"limit":1.0,"other_cpu":[0.34,0.07],"quiet":true,"waited_s":0}},"verdict":"good"}],"run":{"command":"mix defk.examples --bench --all --snapshot","commit":"8fa900b7","elixir":"1.20.2","measured_at":"2026-10-19T11:41:41Z","otp":"29.0.3","quiet_limit":1.0},"status":"ok"}, {"name":"sort_u64_keys","quiet":true,"results":[{"baseline":"optimized","cache":"stored","correct":true,"defk_ms":59.478,"defk_spread_ms":{"max":60.181,"median":60.172,"runs":5},"elixir_ms":355.392,"elixir_spread_ms":{"max":418.879,"median":369.938,"runs":5},"example":"sort_packed","expected":"good","reason":"packed in, packed out: no list is ever built on the defk side","speedup":5.98,"stamp":{"c":{"Keys.Kernels":"a49c2e57190d29d9d5d46c60ecd6535f"},"commit":"684a3592","defk":"0.1.0","measured_at":"2026-10-19T11:59:29Z","quiet":{"limit":1.0,"other_cpu":[0.12,0.04],"quiet":true,"waited_s":0}},"verdict":"good"},{"baseline":"optimized","cache":"reused","correct":true,"defk_ms":94.191,"defk_spread_ms":{"max":118.427,"median":96.442,"runs":5},"elixir_ms":362.119,"elixir_spread_ms":{"max":396.991,"median":371.298,"runs":5},"example":"sort_to_list","expected":"good","measured_at":"2026-10-19T11:41:02Z","reason":"decoding the sorted result to a list costs 40 ms, the margin shrinks to about 2.5x but holds","speedup":3.84,"stamp":{"c":{"Keys.Kernels":"a49c2e57190d29d9d5d46c60ecd6535f"},"commit":"8fa900b7","defk":"0.1.0","measured_at":"2026-10-19T11:41:02Z","quiet":{"limit":1.0,"other_cpu":[0.54,0.22],"quiet":true,"waited_s":6}},"verdict":"good"}],"run":{"command":"mix defk.examples --bench --only codecs,gorilla_encode,monte_carlo_rtp,sort_u64_keys --snapshot","commit":"684a3592","elixir":"1.20.2","measured_at":"2026-10-19T11:59:29Z","otp":"29.0.3","quiet_limit":1.0},"status":"ok"}, + {"name":"tape_csv","quiet":true,"results":[{"baseline":"optimized","cache":"stored","correct":true,"crossover":"1000000 rows","defk_ms":131.174,"defk_spread_ms":{"max":133.365,"median":132.849,"runs":15},"elixir_ms":274.473,"elixir_spread_ms":{"max":279.982,"median":277.864,"runs":15},"example":"tape_csv","expected":"good","native_label":"rusty_csv 0.4.6","native_ms":49.459,"native_speedup":0.38,"native_spread_ms":{"max":52.635,"median":50.216,"runs":15},"realistic_size":"20 MB: a 200k-row customer export","reason":"the kernel cleans 20 MB in 36 ms and two :binary.split passes build the rows, with less garbage than a parser in Elixir; rusty_csv builds them in C, 2.7x faster still","speedup":2.09,"stamp":{"c":{"Tape.Kernels":"9a6a5466f9da258adf6f944d858351d0"},"commit":"71e29777","defk":"0.1.0","measured_at":"2026-10-20T13:41:22Z","quiet":{"limit":1.0,"other_cpu":[0.29,0.82],"quiet":true,"waited_s":0}},"sweep":[{"defk_ms":1.082,"elixir_ms":1.71,"size":"2000 rows","speedup":1.58},{"defk_ms":10.814,"elixir_ms":17.611,"size":"20000 rows","speedup":1.63},{"defk_ms":646.619,"elixir_ms":1524.704,"size":"1000000 rows","speedup":2.36}],"verdict":"good"}],"run":{"command":"mix defk.examples --bench --only tape_csv --snapshot --fresh","commit":"71e29777","elixir":"1.20.2","measured_at":"2026-10-20T13:41:22Z","otp":"29.0.3","quiet_limit":1.0},"status":"ok"}, {"name":"utf8_valid","quiet":true,"results":[{"baseline":"optimized","cache":"stored","correct":true,"crossover":null,"defk_ms":26.992,"defk_spread_ms":{"max":27.922,"median":27.5,"runs":15},"elixir_ms":28.037,"elixir_spread_ms":{"max":29.813,"median":29.156,"runs":15},"example":"utf8_valid","expected":"marginal","realistic_size":"16 MB","reason":"a scalar byte DFA competes with a C BIF; :unicode.characters_to_binary is already C and the DFA is only about 1.07x ahead, so use String.valid? (a SIMD validator is a wrapper job)","speedup":1.04,"stamp":{"c":{"Utf8.Kernels":"5439915fa11707ea174a3f68e6e8af0b"},"commit":"8fa900b7","defk":"0.1.0","measured_at":"2026-10-19T11:41:05Z","quiet":{"limit":1.0,"other_cpu":[0.14,0.38],"quiet":true,"waited_s":0}},"sweep":[{"defk_ms":0.002,"elixir_ms":0.0,"size":1024,"speedup":0.0},{"defk_ms":0.102,"elixir_ms":0.057,"size":65536,"speedup":0.56},{"defk_ms":1.718,"elixir_ms":1.812,"size":1048576,"speedup":1.05},{"defk_ms":27.234,"elixir_ms":29.351,"size":16777216,"speedup":1.08}],"verdict":"marginal"}],"run":{"command":"mix defk.examples --bench --all --snapshot","commit":"8fa900b7","elixir":"1.20.2","measured_at":"2026-10-19T11:41:41Z","otp":"29.0.3","quiet_limit":1.0},"status":"ok"}, {"name":"ws_masking","quiet":true,"results":[{"baseline":"optimized","cache":"stored","correct":true,"crossover":"1024 B","defk_gap_ms":{"max":1.751,"p99":1.751},"defk_ms":1.231,"defk_spread_ms":{"max":1.369,"median":1.287,"runs":15},"elixir_gap_ms":{"max":2.008,"p99":2.008},"elixir_ms":1.793,"elixir_spread_ms":{"max":2.786,"median":2.397,"runs":15},"example":"ws_masking","expected":"marginal","native_gap_ms":{"max":8.345,"p99":8.345},"native_label":"bandit_native 0.1.2 Bandit.Native.mask/2 (Rust, byte at a time, normal scheduler)","native_ms":1.424,"native_speedup":1.16,"native_spread_ms":{"max":1.556,"median":1.461,"runs":15},"realistic_size":"125 B frames (the largest 7-bit length), 10,000 frames per timed run","reason":"a 125 B frame is ~0.13 us of kernel call against the BEAM's already-compiled 32-bit loop; use Bandit's own ws_mask/2 for small frames","speedup":1.46,"stamp":{"c":{"Ws.Defk":"aa97b7005e38e3435e015619bd5af1e8"},"commit":"8fa900b7","defk":"0.1.0","measured_at":"2026-10-19T11:41:21Z","quiet":{"limit":1.0,"other_cpu":[0.34,0.11],"quiet":true,"waited_s":8}},"sweep":[{"defk_ms":1.226,"elixir_ms":1.626,"size":"125 B","speedup":1.33},{"defk_ms":2.176,"elixir_ms":5.996,"size":"1024 B","speedup":2.76},{"defk_ms":1.35,"elixir_ms":4.582,"size":"8192 B","speedup":3.39},{"defk_ms":1.222,"elixir_ms":3.95,"size":"65536 B","speedup":3.23},{"defk_ms":1.203,"elixir_ms":4.109,"size":"1048576 B","speedup":3.42},{"defk_ms":2.547,"elixir_ms":9.134,"size":"16777216 B","speedup":3.59}],"verdict":"marginal"},{"baseline":"optimized","cache":"stored","correct":true,"defk_ms":1.2,"defk_spread_ms":{"max":1.23,"median":1.209,"runs":15},"elixir_ms":4.103,"elixir_spread_ms":{"max":7.095,"median":4.194,"runs":15},"example":"ws_masking_1mib","expected":"good","native_label":"bandit_native 0.1.2 Bandit.Native.mask/2 (Rust, byte at a time, normal scheduler)","native_ms":4.215,"native_speedup":3.51,"native_spread_ms":{"max":4.418,"median":4.38,"runs":15},"realistic_size":"1 MiB frames, 8 per timed run","reason":"the vectorised kernel runs at memory bandwidth (~38 GB/s) while the best Elixir (:crypto.exor/2) and bandit_native's byte-at-a-time loop run at 2 GB/s","speedup":3.42,"stamp":{"c":{"Ws.Defk":"aa97b7005e38e3435e015619bd5af1e8"},"commit":"8fa900b7","defk":"0.1.0","measured_at":"2026-10-19T11:41:21Z","quiet":{"limit":1.0,"other_cpu":[0.11,0.06],"quiet":true,"waited_s":0}},"verdict":"good"}],"run":{"command":"mix defk.examples --bench --all --snapshot","commit":"8fa900b7","elixir":"1.20.2","measured_at":"2026-10-19T11:41:41Z","otp":"29.0.3","quiet_limit":1.0},"status":"ok"} ] diff --git a/examples/tape_csv.exs b/examples/tape_csv.exs new file mode 100644 index 00000000..0deba2b7 --- /dev/null +++ b/examples/tape_csv.exs @@ -0,0 +1,639 @@ +# tape_csv: CSV to rows of binaries through a tape: a kernel scans and +# emits flat field events, Elixir builds the rows +# +# The question (examples/README.md, "Where a wrapper is the right choice"): +# parsers whose product is terms stay out of the catalog because building +# the terms is the cost. What would reopen the row: a tape output (flat +# events of start and length) plus Elixir tree building that beats +# NimbleCSV end to end. This example measures that, for CSV to the lists +# of binaries NimbleCSV.RFC4180.parse_string/2 returns. The numeric form, +# which builds no terms, is numeric_loader.exs. +# +# elixir examples/tape_csv.exs +# +# Pre-registered: marginal (the research: Elixir still builds every +# sub-binary). Measured: good, and the expected verdict below says so. +# +# Two tape forms, both from the same RFC 4180 grammar as NimbleCSV +# (quoted fields with commas, doubled quotes and line breaks; \n or \r\n): +# [i64] tape scan/2: two words per field (start; length plus a "row +# ends" and a "has doubled quotes" flag), sized by a vectorised +# count of ',' and '\n'. Elixir walks it with binary matching +# and takes binary_part of the input per field. +# copy normalize/1: a cleaned copy of the input, separating commas +# as byte 31, row ends as byte 30, quotes removed and doubled +# quotes written once. Elixir splits it twice with +# :binary.split/3 (C), rows then fields. Bytes 30 and 31 in +# the input raise ArgumentError (this form's limit). +# The copy is the faster form and the verdict's defk side. +# +# Elixir baseline: optimised. E1 NimbleCSV.RFC4180 (it already splits +# unquoted runs with :binary.split); E2 lines by :binary.split, a line +# without a quote split on commas, a quoted line walked with :binary.match +# (the baseline: 5-7% faster than E1); E3 E2 with quoted lines split on +# commas and the pieces joined (no gain). Native: rusty_csv 0.4.6 (Rust +# NIF, precompiled; SIMD scan, then sub-binary terms built in C on a dirty +# scheduler). +# +# Agreement: both tape forms, their Elixir mode (`elixir_mode: true`, +# Tape.Kernels.__defk_elixir__/2) and rusty_csv return exactly NimbleCSV's +# rows on 49 inputs (empty input, empty fields and lines, \r\n, quoted +# commas, doubled quotes and line breaks, 1 to 3000 rows); 6 bad inputs (a +# quote inside an unquoted field, an unclosed quote, a byte after a closing +# quote) raise on every side. +# +# Realistic size: 20 MB, a 200k-row customer export (id, name, email, city, +# amount, date, a note quoted in a third of the rows), with a sweep of 2k, +# 20k and 1M rows (100 MB). +# +# Timing: every timed parse runs in a fresh process (spawn, parse, exit), +# as a request handler or a Task per file would. Measured first in one +# long-lived process, best of 15, the answer depended on the heap the +# earlier runs left: with a GC before each run (as Defk.Bench.report/1 +# does) NimbleCSV 201-358 ms and the copy 172-257 ms, 0.81x to 2.08x +# between runs; with no GC between runs NimbleCSV 134-138 ms and the copy +# 96-113 ms. In a fresh process every run grows its heap from nothing, +# and the spread is 2% (median against best). +# +# Measured (M-series Mac, OTP 29, Elixir 1.20, 2026-10-20, fresh process +# per run; the report best of 15, the variant lines best of 5; four runs +# with E2, the last stored and quiet; the others provisional): +# +# 20 MB E1 308-315 ms, E2 274-299 ms, E3 300-305 ms; +# the copy 130.7-141.9 ms 2.09-2.23x over E2 good +# rusty_csv 49.5-53.5 ms: 5.5-5.9x E2, 2.6-2.7x the copy. +# The parts pass 1 4.8-5.1 ms, scan to the [i64] tape 28 ms, rows +# (best of 5) from it 119-127 ms (the tape 152-158 ms end to end); +# the copy 36 ms, its two splits 125-131 ms. +# Sweep 2k rows 1.6-1.7x, 20k 1.6-1.7x, 1M rows (100 MB) 2.4-2.5x. +# +# What it means: a kernel scans CSV at 0.5-0.7 GB/s, and Elixir then +# spends 3.5-4.5x the scan building the rows. The cleaned copy, split by +# the BIFs, beats the best Elixir by about 2.1-2.2x in a fresh process, so a +# tape does reopen the wrapper table's row, narrowly. rusty_csv builds the +# same sub-binaries in C, 2.6x faster again, on a dirty scheduler: where +# the speed matters, it is still the choice. The kernel route needs no +# extra NIF, and its call yields on a normal scheduler. + +Mix.install([ + {:defk, path: Path.expand("..", __DIR__)}, + {:nimble_csv, "~> 1.3"}, + {:rusty_csv, "~> 0.4"} +]) + +defmodule Tape.Source do + # The kernels, as a `use` block for Tape.Kernels, compiled with + # `elixir_mode: true` (native by default; `__defk_elixir__/2` is the + # agreement check's oracle). + defmacro __using__(_) do + quote location: :keep do + # Flags in a field's second tape word, above its length + @esc 1_099_511_627_776 + @eol 2_199_023_255_552 + + # Pass 1: an upper bound on the fields: one per ',' or '\n', plus one. + defk count_separators(s :: [u8]) :: i64 do + for c <- s, reduce: 1 do + acc -> if c == 44 or c == 10, do: wrap64(acc + 1), else: acc + end + end + + # Pass 2: RFC 4180 as NimbleCSV.RFC4180 reads it. Each field is two + # words: its start, and its length plus @esc (a quoted field with + # doubled quotes, to unescape) plus @eol (the last field of a row). + # Rows end in "\n" or "\r\n". A quote inside an unquoted field, or + # anything but a separator after a closing quote, raises ArgumentError. + # States: 0 row start, 1 unquoted field, 2 quoted field, 3 after a + # quote in a quoted field, 4 '\r' after a closing quote, 5 field + # start after ','. + defk scan(s :: [u8], cap :: i64) :: {[i64], i64} do + if length(s) >= @esc, do: raise(ArgumentError, "an input of 1 TiB or more") + tape = alloc(i64, 2 * cap) + n = length(s) + + {st, fs, qe, esc, k} = + for i <- 0..(n - 1)//1, reduce: {0, 0, 0, false, 0} do + {st, fs, qe, esc, k} -> + c = s[i] + + cond do + st == 2 -> + if c == 34, do: {3, fs, i, esc, k}, else: {2, fs, qe, esc, k} + + st == 3 and c == 34 -> + {2, fs, qe, true, k} + + st == 3 and c == 13 -> + {4, fs, qe, esc, k} + + st == 3 or st == 4 -> + if c != 44 and c != 10, do: raise(ArgumentError, "a byte after a closing quote") + + if st == 4 and c != 10, + do: raise(ArgumentError, "a '\\r' after a closing quote") + + tape[2 * k] = fs + w = qe - fs + w = if esc, do: w + @esc, else: w + tape[2 * k + 1] = if c == 10, do: w + @eol, else: w + {if(c == 10, do: 0, else: 5), i + 1, qe, false, k + 1} + + c == 34 -> + if st == 1, do: raise(ArgumentError, "a quote inside an unquoted field") + {2, i + 1, qe, false, k} + + c == 44 -> + tape[2 * k] = if st == 1, do: fs, else: i + tape[2 * k + 1] = if st == 1, do: i - fs, else: 0 + {5, i + 1, qe, false, k + 1} + + c == 10 -> + f = if st == 1, do: fs, else: i + e = if i > f and s[i - 1] == 13, do: i - 1, else: i + tape[2 * k] = f + tape[2 * k + 1] = e - f + @eol + {0, i + 1, qe, false, k + 1} + + true -> + {1, if(st == 1, do: fs, else: i), qe, esc, k} + end + end + + if st == 2 or st == 4, do: raise(ArgumentError, "the input ends inside a quoted field") + + k = + cond do + st == 1 -> + tape[2 * k] = fs + tape[2 * k + 1] = n - fs + @eol + k + 1 + + st == 3 -> + tape[2 * k] = fs + w = qe - fs + tape[2 * k + 1] = if(esc, do: w + @esc, else: w) + @eol + k + 1 + + st == 5 -> + tape[2 * k] = n + tape[2 * k + 1] = @eol + k + 1 + + true -> + k + end + + {tape, k} + end + + # The other tape form: a cleaned copy. Separating commas become 31 + # (US), row ends 30 (RS, also after the last row), the quotes around + # a quoted field go and a doubled quote is written once; a "\r" + # before "\n" outside quotes goes. Returns {buffer, length}. A 30 or + # 31 byte in the input raises ArgumentError (this form's limit); the + # rest of the grammar is scan/2's. States as in scan/2. + defk normalize(s :: [u8]) :: {[u8], i64} do + n = length(s) + out = alloc(u8, n + 1) + + {st, j} = + for i <- 0..(n - 1)//1, reduce: {0, 0} do + {st, j} -> + c = s[i] + if c == 30 or c == 31, do: raise(ArgumentError, "a 30 or 31 byte in the input") + + cond do + st == 2 -> + if c == 34 do + {3, j} + else + out[j] = c + {2, j + 1} + end + + st == 3 and c == 34 -> + out[j] = 34 + {2, j + 1} + + st == 3 and c == 13 -> + {4, j} + + st == 3 or st == 4 -> + if c != 44 and c != 10, do: raise(ArgumentError, "a byte after a closing quote") + + if st == 4 and c != 10, + do: raise(ArgumentError, "a '\\r' after a closing quote") + + out[j] = if c == 10, do: 30, else: 31 + {if(c == 10, do: 0, else: 5), j + 1} + + c == 34 -> + if st == 1, do: raise(ArgumentError, "a quote inside an unquoted field") + {2, j} + + c == 44 -> + out[j] = 31 + {5, j + 1} + + c == 10 -> + out[j] = 30 + {0, j + 1} + + c == 13 and i + 1 < n and s[i + 1] == 10 -> + {st, j} + + true -> + out[j] = c + {1, j + 1} + end + end + + if st == 2 or st == 4, do: raise(ArgumentError, "the input ends inside a quoted field") + closes = st == 1 or st == 3 or st == 5 + if closes, do: out[j] = 30 + {out, if(closes, do: j + 1, else: j)} + end + end + end +end + +defmodule Tape.Kernels do + use Defk, elixir_mode: true + use Tape.Source +end + +defmodule Tape.Build do + import Bitwise + + @len 1_099_511_627_775 + @esc 1_099_511_627_776 + @eol 2_199_023_255_552 + + # The rows NimbleCSV.RFC4180.parse_string(csv, skip_headers: false) + # returns: lists of sub-binaries of csv, a quoted field with doubled + # quotes unescaped into a new binary. + def rows(csv, {tape, k}), do: build(binary_part(tape, 0, 16 * k), csv, [], []) + + defp build(<>, csv, row, acc) do + f = binary_part(csv, s, w &&& @len) + f = if (w &&& @esc) != 0, do: :binary.replace(f, "\"\"", "\"", [:global]), else: f + + if (w &&& @eol) != 0, + do: build(rest, csv, [], [:lists.reverse([f | row]) | acc]), + else: build(rest, csv, [f | row], acc) + end + + defp build(<<>>, _csv, [], acc), do: :lists.reverse(acc) +end + +defmodule Tape.Calls do + def parse(csv), do: Tape.Build.rows(csv, scan(csv)) + def scan(csv), do: Tape.Kernels.scan(csv, Tape.Kernels.count_separators(csv)) + + # The cleaned copy, split by the :binary BIFs: rows at 30, fields at 31. + def parse_normalized(csv), do: split_normalized(Tape.Kernels.normalize(csv)) + + def split_normalized({_out, 0}), do: [] + + def split_normalized({out, j}) do + for row <- :binary.split(binary_part(out, 0, j - 1), <<30>>, [:global]), + do: :binary.split(row, <<31>>, [:global]) + end +end + +defmodule Tape.Elixir do + # The :binary BIFs do the scanning. Lines by :binary.split; a line + # without a quote is split on commas. A line with one is joined with the + # next while a quote is open, then E2 walks it field by field with + # :binary.match, and E3 splits it on commas and joins the pieces of each + # quoted field. The same rows as NimbleCSV.RFC4180. + def parse(csv), do: parse(csv, :walk) + def parse_merge(csv), do: parse(csv, :merge) + + defp parse("", _mode), do: [] + + # The last line is open when the input does not end in "\n": NimbleCSV + # keeps a "\r" there. + defp parse(csv, mode) do + lines = :binary.split(csv, "\n", [:global]) + + if :binary.last(csv) == ?\n, + do: rows(:lists.droplast(lines), true, mode, []), + else: rows(lines, false, mode, []) + end + + defp rows([], _closed, _mode, acc), do: :lists.reverse(acc) + + defp rows([l | rest], closed, mode, acc) do + case :binary.match(l, "\"") do + :nomatch -> + rows(rest, closed, mode, [:binary.split(strip_cr(l, rest, closed), ",", [:global]) | acc]) + + _ -> + record(l, rest, closed, mode, acc) + end + end + + # join lines until the record's quotes balance + defp record(l, rest, closed, mode, acc) do + if rem(length(:binary.matches(l, "\"")), 2) == 0 do + r = strip_cr(l, rest, closed) + + row = + if mode == :walk, do: fields(r, 0, []), else: merge(:binary.split(r, ",", [:global]), []) + + rows(rest, closed, mode, [row | acc]) + else + case rest do + [next | rest] -> record(<>, rest, closed, mode, acc) + [] -> raise ArgumentError, "the input ends inside a quoted field" + end + end + end + + # E3: pieces between commas; a piece that opens a quote takes the next + # pieces until its quotes balance and it ends in a quote + defp merge([], acc), do: :lists.reverse(acc) + defp merge([<> = p | rest], acc), do: collect(p, rest, acc) + + defp merge([p | rest], acc) do + if :binary.match(p, "\"") != :nomatch, + do: raise(ArgumentError, "a quote inside an unquoted field") + + merge(rest, [p | acc]) + end + + defp collect(buf, rest, acc) do + n = byte_size(buf) + + if n >= 2 and :binary.last(buf) == ?" and rem(length(:binary.matches(buf, "\"")), 2) == 0 do + inner = binary_part(buf, 1, n - 2) + + case :binary.match(inner, "\"") do + :nomatch -> + merge(rest, [inner | acc]) + + _ -> + # a quote that is not doubled ends the field early: NimbleCSV raises + if :binary.match(:binary.replace(inner, "\"\"", "", [:global]), "\"") != :nomatch, + do: raise(ArgumentError, "a byte after a closing quote") + + merge(rest, [:binary.replace(inner, "\"\"", "\"", [:global]) | acc]) + end + else + case rest do + [p | rest] -> collect(<>, rest, acc) + [] -> raise ArgumentError, "a byte after a closing quote" + end + end + end + + defp strip_cr(l, [], false), do: l + + defp strip_cr(l, _rest, _closed) do + n = byte_size(l) + if n > 0 and :binary.at(l, n - 1) == ?\r, do: binary_part(l, 0, n - 1), else: l + end + + defp fields(r, p, acc) do + n = byte_size(r) + + if p < n and :binary.at(r, p) == ?" do + {f, q} = quoted(r, p + 1, p + 1, false) + + cond do + q == n -> :lists.reverse([f | acc]) + :binary.at(r, q) == ?, -> next(r, q + 1, [f | acc]) + true -> raise ArgumentError, "a byte after a closing quote" + end + else + case :binary.match(r, ",", scope: {p, n - p}) do + {c, 1} -> next(r, c + 1, [unquoted(r, p, c - p) | acc]) + :nomatch -> :lists.reverse([unquoted(r, p, n - p) | acc]) + end + end + end + + # after a ',' a field always follows, empty at the end of the record + defp next(r, p, acc) when p == byte_size(r), do: :lists.reverse(["" | acc]) + defp next(r, p, acc), do: fields(r, p, acc) + + defp unquoted(r, p, len) do + f = binary_part(r, p, len) + + if :binary.match(f, "\"") != :nomatch, + do: raise(ArgumentError, "a quote inside an unquoted field") + + f + end + + # from s, the field's first byte; returns {field, index after the closing quote} + defp quoted(r, s, p, esc) do + case :binary.match(r, "\"", scope: {p, byte_size(r) - p}) do + :nomatch -> + raise ArgumentError, "the input ends inside a quoted field" + + {q, 1} -> + if q + 1 < byte_size(r) and :binary.at(r, q + 1) == ?" do + quoted(r, s, q + 2, true) + else + f = binary_part(r, s, q - s) + {if(esc, do: :binary.replace(f, "\"\"", "\"", [:global]), else: f), q + 1} + end + end + end +end + +defmodule Tape.Data do + # A customer export: id, name, email, city, amount, signup date, and a + # free-text note that is quoted in a third of the rows (commas inside), + # with doubled quotes in a tenth and a line break in one row of 200. + @first ~w(Anna Bo Carlos Dana Emil Fatima Gustav Hana Ivan Jade Karl Lena) + @last ~w(Svensson Okafor Lindqvist Moreau Tanaka Novak Haddad Berg Costa Ito) + @city ["Stockholm", "Malmö", "Göteborg", "Uppsala", "Valletta", "London", "São Paulo", "Kraków"] + @words ~w(called about the invoice wants a refund new address prefers email VIP late payment) + + def csv(rows, seed \\ 1) do + :rand.seed(:exsss, {seed, 3, 5}) + + IO.iodata_to_binary( + for id <- 1..rows//1 do + fst = Enum.random(@first) + lst = Enum.random(@last) + + [ + Integer.to_string(id), + ?,, + fst, + " ", + lst, + ?,, + String.downcase(fst), + ?., + String.downcase(lst), + "@example.com,", + Enum.random(@city), + ?,, + :erlang.float_to_binary(:rand.uniform() * 5000, decimals: 2), + ?,, + "2026-", + pad(:rand.uniform(12)), + ?-, + pad(:rand.uniform(28)), + ?,, + note(), + "\r\n" + ] + end + ) + end + + defp pad(n) when n < 10, do: [?0, Integer.to_string(n)] + defp pad(n), do: Integer.to_string(n) + + defp note do + words = Enum.join(Enum.take_random(@words, :rand.uniform(6)), " ") + + case :rand.uniform(200) do + r when r <= 20 -> [?", words, ", said \"\"ok\"\"", ?"] + r when r <= 21 -> [?", words, "\nsecond line", ?"] + r when r <= 67 -> [?", words, ", ", words, ?"] + _ -> words + end + end +end + +alias Tape.{Calls, Data, Kernels} + +nimble = &NimbleCSV.RFC4180.parse_string(&1, skip_headers: false) +rusty = &RustyCSV.RFC4180.parse_string(&1, skip_headers: false) + +# ---- agreement ---- + +raises? = fn f -> + try do + f.() + false + rescue + _ -> true + end +end + +oracle = fn csv -> + Tape.Build.rows(csv, Kernels.__defk_elixir__(:scan, [csv, Kernels.count_separators(csv)])) +end + +cases = + [ + "", + "a", + "a,b", + "a,b\n", + "a,b\r\n", + "a,b\n\nc,d\n", + "a,\n,b\n", + ",", + ",\n", + "\n", + "\r\n", + "\"a,1\",\"x\"\"y\",\"\"\n", + "\"q\"", + "\"q\"\r\n", + "\"multi\nline\",x\n", + "\"\"\"\"", + "a\r", + "x,\"y\"", + Data.csv(3000, 9) + ] ++ for(r <- 1..30, do: Data.csv(r, r)) + +agree = + Enum.all?(cases, fn c -> + (p = Calls.parse(c)) == nimble.(c) and p == oracle.(c) and p == Calls.parse_normalized(c) and + p == Calls.split_normalized(Kernels.__defk_elixir__(:normalize, [c])) and + p == Tape.Elixir.parse(c) and p == Tape.Elixir.parse_merge(c) + end) + +bad = ["a\"b,c\n", "\"a\"b,c\n", "\"open\n", "\"open", "\"a\"\rb", "x,\"y\"z"] + +bad_ok = + Enum.all?(bad, fn c -> + raises?.(fn -> Calls.parse(c) end) and raises?.(fn -> oracle.(c) end) and + raises?.(fn -> nimble.(c) end) and raises?.(fn -> Calls.parse_normalized(c) end) and + raises?.(fn -> Tape.Elixir.parse(c) end) and raises?.(fn -> Tape.Elixir.parse_merge(c) end) + end) + +csv = Data.csv(200_000) +rows = nimble.(csv) +rusty_agree = rusty.(csv) == rows + +IO.puts(""" +agreement: tape == NimbleCSV and Elixir mode on #{length(cases)} inputs #{agree}, \ +bad input raises on every side #{bad_ok} (#{length(bad)} cases), rusty_csv agrees #{rusty_agree} +""") + +# Every timed parse runs in a fresh process, as a request handler would, +# and the time includes its exit. Building 1.4M terms in one long-lived +# process gave answers that depended on the heap the earlier runs left: +# NimbleCSV 134-358 ms, the copy 96-264 ms, the verdict 0.8x to 2.1x. +fresh = fn f -> + fn -> + {_pid, ref} = spawn_monitor(fn -> f.() end) + + receive do + {:DOWN, ^ref, :process, _, :normal} -> :ok + {:DOWN, ^ref, :process, _, reason} -> exit(reason) + end + end +end + +best = fn f, runs -> + g = fresh.(f) + for(_ <- 1..runs, do: elem(:timer.tc(g), 0)) |> Enum.min() |> Kernel./(1000) |> Float.round(3) +end + +tape = Calls.scan(csv) +norm = Kernels.normalize(csv) + +IO.puts( + "csv, #{Float.round(byte_size(csv) / 1_000_000, 1)} MB (#{length(rows)} rows x 7 fields), best of 5:" +) + +for {label, f} <- [ + {"E1 NimbleCSV.RFC4180.parse_string", fn -> nimble.(csv) end}, + {"E2 :binary.split, quoted lines walked", fn -> Tape.Elixir.parse(csv) end}, + {"E3 :binary.split, quoted pieces joined", fn -> Tape.Elixir.parse_merge(csv) end}, + {"rusty_csv parse_string (Rust NIF)", fn -> rusty.(csv) end}, + {"defk pass 1 (count separators)", fn -> Kernels.count_separators(csv) end}, + {"defk pass 2 (scan to tape)", fn -> Kernels.scan(csv, elem(tape, 1) + 1) end}, + {"Elixir rows from the tape", fn -> Tape.Build.rows(csv, tape) end}, + {"tape end to end", fn -> Calls.parse(csv) end}, + {"defk normalize (cleaned copy)", fn -> Kernels.normalize(csv) end}, + {"Elixir rows from the copy (2 splits)", fn -> Calls.split_normalized(norm) end}, + {"cleaned copy end to end", fn -> Calls.parse_normalized(csv) end} + ] do + IO.puts(" #{String.pad_trailing(label, 36)} #{best.(f, 5)} ms") +end + +# ---- the verdict: the cleaned copy (the faster tape form) against +# NimbleCSV, with rusty_csv's NIF beside it ---- + +sweep = fn rows -> + c = Data.csv(rows, 17) + {fresh.(fn -> Tape.Elixir.parse(c) end), fresh.(fn -> Calls.parse_normalized(c) end)} +end + +Defk.Bench.report( + name: "tape_csv", + expected: :good, + baseline: :optimized, + runs: 15, + realistic_size: "20 MB: a 200k-row customer export", + elixir: fresh.(fn -> Tape.Elixir.parse(csv) end), + defk: fresh.(fn -> Calls.parse_normalized(csv) end), + native: {fresh.(fn -> rusty.(csv) end), "rusty_csv 0.4.6"}, + check: fn -> agree and bad_ok and rusty_agree and Calls.parse_normalized(csv) == rows end, + sweep: for(r <- [2_000, 20_000, 1_000_000], do: Tuple.insert_at(sweep.(r), 0, "#{r} rows")), + sweep_runs: 3, + reason: + "the kernel cleans 20 MB in 36 ms and two :binary.split passes build the rows, with less garbage than a parser in Elixir; rusty_csv builds them in C, 2.7x faster still" +)