Skip to content

defk-9p6.21: numeric_loader, a fused CSV and JSON number loader into packed f64s - #74

Merged
DanielLuning merged 3 commits into
mainfrom
defk-9p6.21/numeric-loader
Oct 20, 2026
Merged

DanielLuning merged 3 commits into
mainfrom
defk-9p6.21/numeric-loader

Conversation

@DanielLuning

Copy link
Copy Markdown
Owner

What changed

  • New example examples/numeric_loader.exs: a numeric CSV (rows of -?[0-9]+\.[0-9]+ fields) and a flat JSON array of numbers, loaded straight into packed [f64]. Two passes: a vectorised count of newlines or commas, then one state-machine parse into an alloc of that size. The grammar is written down: at most 18 digits and a mantissa of at most 2^53, so one division gives the bits of String.to_float/1. Everything else raises ArgumentError.
  • Elixir variants, all timed in each run: NimbleCSV + String.to_float, :binary.split + binary_to_float, :binary.match + binary_to_float (header only), and a binary-matching parser with the kernel's algorithm (the baseline). JSON: Jason, OTP's :json, and the same binary-matching parser.
  • examples/scoreboard.json: the new row, quiet, stamped at 0d6e7c7 (--snapshot --fresh from a clean tree). .gitleaksignore: the sort_u64_keys pin moves from line 34 to 35, as the new row sits above it.
  • examples/_tables.exs, both READMEs (regenerated), an example bullet, and a note in the wrapper table's parser row: a packed numeric load is an exception like BPE ids.
  • docs/performance.md 2.18: what the Elixir side tried, the cost of the count pass, and the wrap64 count that vectorises.

Results

best Elixir defk speedup verdict
CSV, 26 MB (400k rows x 8) 138.8-143.7 ms 50.5-53.0 ms 2.68-2.78x good
JSON, 2.2 MB (153,600 floats) 9.0-12.4 ms 2.08-2.34 ms 3.90-5.87x good
  • Eight runs. The stored run passed the quiet test for both reports. The CSV numbers of the other runs are provisional (up to 1.9 cores of other work after the measurement).
  • The JSON Elixir side is bimodal: its best of 15 is 9-12 ms, its median 18-20 ms, depending on whether a run escapes GC.
  • The count pass is 6% (CSV) and 11% (JSON) of the kernel's time, under the ticket's 20% line, so no growable output construct is needed.
  • The binary-matching Elixir parser that appends to one binary is 5.7-5.9x NimbleCSV + String.to_float; with an iolist it was 1.7-1.9x slower.
  • One kernel shape was tried and dropped: a loop over the fields with an Enum.reduce_while per field. It was 15% slower.

How it was checked

  • mix ci green in the worktree on the Mac (2305 passed). test/scoreboard_doc_test.exs checks the generated tables.

  • Agreement: every Elixir side, the kernels and their Elixir mode give the same bytes on 26 CSV and 28 JSON inputs. 411 edge fields are bit-exact with String.to_float/1. 23 bad CSV and 26 bad JSON inputs raise on every side that has the grammar.

  • mix defk.examples --stale lists the new row as current.

  • Touches a promised surface (kernel subset, runtime contract, public API, error codes)? Then a CHANGELOG.md entry, and a deprecation for a rename or removal (docs/guides/stability.md).

  • Touches test/stability/? A frozen file is never edited; a release adds a directory (docs/agents/stability.md).

…packed f64s

Two passes (count newlines or commas, then parse into one alloc) with a
written-down grammar: -?[0-9]+\.[0-9]+ for CSV fields, JSON numbers
without exponents, at most 18 digits and a mantissa of at most 2^53, so
one division gives the bits of String.to_float/1. Everything else raises.
Good on both: CSV 26 MB 2.68-2.75x over a binary-matching Elixir parser
(itself 5.7-5.9x NimbleCSV + String.to_float), JSON 2.2 MB of embedding
floats 3.90-4.32x over the same parser, Jason and OTP's :json. The count
pass is 6-11% of the time, so no growable output is needed.
… and ranges

mix defk.examples --bench --only numeric_loader --snapshot --fresh from a
clean tree: both reports quiet, CSV 2.74x, JSON 4.03x. The header's
ranges cover all eight runs (CSV 2.68-2.78x, JSON 3.90-5.87x: the Elixir
side's best of 15 depends on whether a run escapes GC). The header keeps
its line count, so the stamped C is the committed kernels'. The
sort_u64_keys gitleaks pin moves to line 35.
@DanielLuning
DanielLuning merged commit 4ecc047 into main Oct 20, 2026
2 checks passed
@DanielLuning
DanielLuning deleted the defk-9p6.21/numeric-loader branch October 20, 2026 19:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant