defk-9p6.30: tape_csv, CSV to rows through a kernel tape; the cleaned copy is 2.1-2.2x the best Elixir

README.md

@@ -43,6 +43,7 @@ Speedups over an optimised Elixir baseline, measured end to end on Apple M4 Pro
 | ✅ | base64, hex and base58 codecs (4.2× the fast64 NIF); Snappy and LZ4 block decode on untrusted input | 5.1–6.1× |
 | ✅ | sorting packed keys, nearest of 500k locations, Monte Carlo slot simulation with `:rand` bit for bit, Gorilla encoding | 2.4–6.0× |
 | ✅ | a numeric CSV or a JSON array of numbers straight into packed f64s (NimbleCSV and Jason stay for anything else) | 2.7× |
+| ✅ | CSV to NimbleCSV's rows through a kernel that cleans the separators for `:binary.split` (rusty_csv's NIF is faster still) | 2.1× |
 | ✅ | a fuzzy-search web service, Myers bit-parallel on both sides: `/health` latency stays at Elixir's while a blocking NIF raises it | [its own measurement](https://github.com/DanielLuning/defk/blob/main/examples/fuzzy_search/README.md) |
 | 🟡 | a dictionary-wide Levenshtein batch over all schedulers: Myers bit-parallel in Elixir is close | 1.5× |
 | 🟡 | small WebSocket frames, UTF-8 validation, substring search: the BIFs are native already | 1.0–1.5× |

docs/performance.md

@@ -2917,6 +2917,36 @@ output array would).
 - **Source:** [`bench/port_roundtrip.exs`](https://github.com/DanielLuning/defk/blob/main/bench/port_roundtrip.exs), [`bench/port_echo.c`](https://github.com/DanielLuning/defk/blob/main/bench/port_echo.c), `defk-7dm`.
 - **Status:** one quiet run; rerun with `defk-sgw.4`'s snapshot if the verdict cites it.
 
+### 7.11 CSV to terms: a kernel that cleans the separators for `:binary.split` is 2.1-2.2x the best Elixir; rusty_csv builds the sub-binaries in C, 2.6x faster again
+
+A kernel cannot build terms, but it can make the input easy for the BIFs:
+a cleaned copy with byte 31 between fields and byte 30 after each row,
+quotes removed, that two `:binary.split/3` passes turn into NimbleCSV's
+rows. A tape of field events walked in Elixir is slower: reading the tape
+costs as much as the splits.
+
+- **Numbers** (20 MB, a 200k-row export with quoted fields; a fresh process
+  per parse): NimbleCSV 308-315 ms; the best Elixir (lines by
+  `:binary.split`, unquoted lines split on commas, quoted lines walked with
+  `:binary.match`) 274-299 ms; the cleaned copy 131-142 ms end to end
+  (2.09-2.23x; the copy 36 ms, the splits 125-131 ms); an `[i64]` tape of start
+  and length per field 152-158 ms (scan 28 ms, rows from it 119-127 ms;
+  only walking its 1.4M entries took 39 ms); rusty_csv 0.4.6 49.5-53.5 ms.
+- **Heap state decides term-building benchmarks:** timed in one long-lived
+  process, best of 15, the same code gave NimbleCSV 134-358 ms and the copy
+  96-264 ms, 0.81x to 2.08x between runs, with or without a GC before each
+  run. A fresh process per run grows its heap from nothing every time; the
+  spread was 2% (median against best).
+- **What it means:** the 13x gap between scanning and term creation that
+  `string-workloads.md` measured holds for Elixir, not for C: rusty_csv's
+  sub-binaries cost little. For CSV to rows, rusty_csv is the fastest; a
+  kernel's cleaned copy is a middle option without another NIF, and yields
+  on a normal scheduler. JSON to terms is not measured.
+- **Conditions:** M-series Mac, OTP 29, Elixir 1.20.2; four runs with a
+  fresh process, five before in a long-lived one.
+- **Source:** [`tape_csv.exs`](https://github.com/DanielLuning/defk/blob/main/examples/tape_csv.exs), `defk-9p6.30`.
+- **Status:** measured (the stored run, quiet); the rest provisional.
+
 ---
 
 ## 8. Myths we disproved and open questions

examples/README.md

@@ -43,7 +43,7 @@ script tried and what to take from it; the numbers behind them are in each
 script's header.
 
 <!-- scoreboard:start -->
-Measured on Apple M4 Pro (12 logical cores) in 3 runs, from 2026-10-19 to 2026-10-20. The newest at commit `0d6e7c78`, OTP 29.0.3, Elixir 1.20.2: `mix defk.examples --bench --only numeric_loader --snapshot --fresh`. A quiet measurement: other processes used at most 1.0 cores before and after it. A row from an earlier run says when it was measured, and at which commit.
+Measured on Apple M4 Pro (12 logical cores) in 4 runs, from 2026-10-19 to 2026-10-20. The newest at commit `71e29777`, OTP 29.0.3, Elixir 1.20.2: `mix defk.examples --bench --only tape_csv --snapshot --fresh`. A quiet measurement: other processes used at most 1.0 cores before and after it. A row from an earlier run says when it was measured, and at which commit.
 
 | | Example | The job | defk vs best Elixir | vs the package's NIF | In one line |
 |---|---|---|---|---|---|
@@ -67,9 +67,10 @@ Measured on Apple M4 Pro (12 logical cores) in 3 runs, from 2026-10-19 to 2026-1
 | ✅ | [`monte_carlo_rtp`](monte_carlo_rtp.exs) (measured 2026-10-19 at `684a3592`) | 100M slot spins on all schedulers | **3.9×** | – | ≈1× at 1M spins: short kernel chunks stay on few schedulers |
 | ✅ | [`per_pair_distance`](per_pair_distance.exs) (measured 2026-10-19 at `8fa900b7`) | Levenshtein on 50k short pairs, one call each | **3.6×** | – | stay in Elixir for very short strings |
 | ✅ | [`geo_nearest`](geo_nearest.exs) (measured 2026-10-19 at `8fa900b7`) | 10 nearest of 500k locations | **3.2×** | – | 14× in one process; a grid prefilter helps both sides |
-| ✅ | [`numeric_loader`](numeric_loader.exs) | numeric CSV of 26 MB into packed f64s | **2.7×** | – | a JSON array of numbers 4.0×; a fixed number grammar, and NimbleCSV or Jason for anything else |
+| ✅ | [`numeric_loader`](numeric_loader.exs) (measured 2026-10-20 at `0d6e7c78`) | numeric CSV of 26 MB into packed f64s | **2.7×** | – | a JSON array of numbers 4.0×; a fixed number grammar, and NimbleCSV or Jason for anything else |
 | ✅ | [`cdcl_sat_per_call`](cdcl_sat_per_call.exs) (measured 2026-10-19 at `8fa900b7`) | one SAT solve per call, 26–28 vars | **2.6×** | 1.6× picosat | ≈1× on tiny formulas; batch the solves when one question makes several |
 | ✅ | [`gorilla_encode`](gorilla_encode.exs) (measured 2026-10-19 at `684a3592`) | Gorilla encoding of 100-point blocks | **2.4×** | 4.2× gorilla_stream's NIF | close to the bar at this size; 4.1× at 1M points |
+| ✅ | [`tape_csv`](tape_csv.exs) | CSV of 20 MB to NimbleCSV's rows, a fresh process per parse | **2.1×** | 0.38× rusty_csv | in a long-lived process 0.8–2.1× run to run; an [i64] tape of field events is slower than the cleaned copy |
 | ✅ | [`fuzzy_search/`](fuzzy_search) | a search service over 234k words behind Bandit | its own measurement, see its README | – | Myers on both sides; `/health` latency stays at Elixir's while a blocking NIF raises it |
 | 🟡 | [`batched_levenshtein`](batched_levenshtein.exs) (measured 2026-10-19 at `8fa900b7`) | one query against 234k words, 12 chunks each side | 1.5× | 13× ex_strsim per pair | Myers bit-parallel in plain Elixir is close; use it |
 | 🟡 | [`ws_masking`](ws_masking.exs) (measured 2026-10-19 at `8fa900b7`) | mask 125-byte WebSocket frames | 1.5× | 1.2× bandit_native | the call is the cost; 3.4× at 1 MiB, crossover 1024 B |
@@ -193,6 +194,13 @@ has them in full.
   5.7–5.9× faster than NimbleCSV with `String.to_float`; the kernel is 2.7×
   that on CSV, and about 4× on JSON against Jason, OTP's `:json` and the
   same parser.
+- [`tape_csv.exs`](tape_csv.exs): CSV to NimbleCSV's rows of binaries
+  through a kernel tape. An `[i64]` tape of field events loses to a
+  cleaned copy (separators as bytes 30 and 31) that two `:binary.split`
+  passes turn into rows. In a fresh process per parse the copy is 2.1–2.2× the
+  best Elixir; rusty_csv's NIF builds the same sub-binaries in C, 2.6×
+  faster again. Timed in one long-lived process, the verdict moved from
+  0.8× to 2.1× run to run with the heap the earlier runs left.
 - [`lz4_snappy_decode.exs`](lz4_snappy_decode.exs): Snappy and LZ4 block
   decode from scratch on untrusted input, against snappyer, lz4_erl and
   nimble_lz4. Against the NIFs as installed from Hex (built with no `-O`)
@@ -298,7 +306,7 @@ Nothing here is excluded on principle: each row names the condition that would r
 | Job | Wrapped today | Why not from scratch | What would change it |
 |---|---|---|---|
 | Signing, key exchange, secret-dependent crypto (secp256k1, ed25519, X25519, AEAD) | libsodium, secp256k1, ring via Rustler; `:crypto` | The one rule says nothing about timing: clang `-O3` may turn a select into a branch or a table lookup into a secret-dependent access, and a process dump or yielded frame holds key material. Public-input verification is allowed but gains little. | A constant-time mode: a restricted subset checked at compile time, codegen verified with dudect or ctgrind in CI, frame zeroisation, and an external audit. Until then, `:crypto`. |
-| Parsers that produce trees or term streams (HTML, Markdown, tree-sitter, JSON/CSV to terms, protobuf, msgpack) | html5ever, comrak, tree-sitter, jsont, rusty_csv, protox | The product is BEAM terms and building them is the cost: `docs/research/string-workloads.md` measured a 13x gap between scanning and term creation, and a NIF builds terms serially at the same price. A kernel cannot build terms, and if it could it would be no faster. | A "tape" output (flat `[i64]` of kind/start/len events, of which a fused CSV parser is one form), plus a benchmark showing tape plus Elixir tree building beats Jason or NimbleCSV end to end. BPE tokenisation to `[i64]` ids is the exception: its product is packed, and so is a numeric CSV or JSON number array loaded into f64s (`numeric_loader`, good: about 2.7× and 4×). |
+| Parsers that produce trees or term streams (HTML, Markdown, tree-sitter, JSON/CSV to terms, protobuf, msgpack) | html5ever, comrak, tree-sitter, jsont, rusty_csv, protox | The product is BEAM terms and building them is the cost: `docs/research/string-workloads.md` measured a 13x gap between scanning and term creation. [`tape_csv`](tape_csv.exs) measured the tape route for CSV: a kernel cleans 20 MB in 36 ms, and two `:binary.split` passes then take 125–131 ms to build the rows, 2.1–2.2× the best Elixir in a fresh process per parse (in one long-lived process 0.8–2.1×, with the heap). rusty_csv builds the same sub-binaries in C, 2.6× faster again: a NIF builds terms far cheaper than Elixir does. A kernel cannot build terms. | Reopened narrowly for CSV to rows (`tape_csv`, good) where an extra NIF is not wanted; rusty_csv stays the fastest. JSON to terms is not measured. BPE tokenisation to `[i64]` ids is the exception: its product is packed, and so is a numeric CSV or JSON number array loaded into f64s (`numeric_loader`, good: about 2.7× and 4×). |
 | Regular expressions | `:re` (PCRE in C), Rust regex via Rustler | `:re` is already native and compiled once per pattern; a kernel cannot take a pattern as data without becoming an interpreter, and fixed-string search is `:binary.match`. | A specific pattern class with a known kernel form (glob, or a simple byte DFA compiled at Elixir compile time) and a benchmark showing `:re` loses on it. |
 | Large codecs and encoders (brotli, zstd encode, audio and video, PDF/typst rendering) | brotli, zstd, membrane_* FFmpeg/opus bindings, typst | The cost is spec surface and tuning, not the loop: 10^4-10^5 lines with SIMD everywhere. A kernel would be a worse implementation of a solved problem. | A narrow, named need: one compression level with fixed tables (zstd level 1 is the lz4-encode ticket plus FSE) or a decoder-only requirement for untrusted input. Each reopens one row, not the category. |
 | Systems: databases, dataframes, tensor compilers (SQLite, DuckDB, Polars/Explorer, XLA/EXLA, LibTorch) | exqlite, duckdbex, explorer, exla, torchx | The value is the library: query planners, columnar engines, autodiff, device backends. defk is a kernel, not a system. | Interop, not rewriting: zero-copy views over their packed buffers so a per-element loop beside them runs as a kernel. `docs/research/nx-boundary.md` says where that line is; file an interop ticket when a workload asks. |

examples/_tables.exs

@@ -160,6 +160,11 @@
       job: "sort 1M packed keys",
       line: "{speedup:sort_to_list} when the caller needs a list back"
     },
+    "tape_csv" => %{
+      job: "CSV of 20 MB to NimbleCSV's rows, a fresh process per parse",
+      native: "{native} rusty_csv",
+      line: "in a long-lived process 0.8–2.1× run to run; an [i64] tape of field events is slower than the cleaned copy"
+    },
     "utf8_valid" => %{
       job: "validate 16 MB of UTF-8",
       line: "`String.valid?` is C already"
@@ -227,6 +232,11 @@
         "a numeric CSV or a JSON array of numbers straight into packed f64s (NimbleCSV and Jason stay for anything else)",
       examples: ["numeric_loader"]
     },
+    %{
+      text:
+        "CSV to NimbleCSV's rows through a kernel that cleans the separators for `:binary.split` (rusty_csv's NIF is faster still)",
+      examples: ["tape_csv"]
+    },
     %{
       text:
         "a fuzzy-search web service, Myers bit-parallel on both sides: `/health` latency stays at Elixir's while a blocking NIF raises it",