Repository navigation
defk-9p6.21: numeric_loader, a fused CSV and JSON number loader into packed f64s - #74
Merged
Merged
Conversation
…packed f64s Two passes (count newlines or commas, then parse into one alloc) with a written-down grammar: -?[0-9]+\.[0-9]+ for CSV fields, JSON numbers without exponents, at most 18 digits and a mantissa of at most 2^53, so one division gives the bits of String.to_float/1. Everything else raises. Good on both: CSV 26 MB 2.68-2.75x over a binary-matching Elixir parser (itself 5.7-5.9x NimbleCSV + String.to_float), JSON 2.2 MB of embedding floats 3.90-4.32x over the same parser, Jason and OTP's :json. The count pass is 6-11% of the time, so no growable output is needed.
… and ranges mix defk.examples --bench --only numeric_loader --snapshot --fresh from a clean tree: both reports quiet, CSV 2.74x, JSON 4.03x. The header's ranges cover all eight runs (CSV 2.68-2.78x, JSON 3.90-5.87x: the Elixir side's best of 15 depends on whether a run escapes GC). The header keeps its line count, so the stamped C is the committed kernels'. The sort_u64_keys gitleaks pin moves to line 35.
DanielLuning
pushed a commit
that referenced
this pull request
Oct 20, 2026
2 tasks
DanielLuning
pushed a commit
that referenced
this pull request
Oct 20, 2026
Merged
2 tasks
…oader # Conflicts: # examples/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
examples/numeric_loader.exs: a numeric CSV (rows of-?[0-9]+\.[0-9]+fields) and a flat JSON array of numbers, loaded straight into packed[f64]. Two passes: a vectorised count of newlines or commas, then one state-machine parse into an alloc of that size. The grammar is written down: at most 18 digits and a mantissa of at most 2^53, so one division gives the bits ofString.to_float/1. Everything else raisesArgumentError.String.to_float,:binary.split+binary_to_float,:binary.match+binary_to_float(header only), and a binary-matching parser with the kernel's algorithm (the baseline). JSON: Jason, OTP's:json, and the same binary-matching parser.examples/scoreboard.json: the new row, quiet, stamped at 0d6e7c7 (--snapshot --freshfrom a clean tree)..gitleaksignore: the sort_u64_keys pin moves from line 34 to 35, as the new row sits above it.examples/_tables.exs, both READMEs (regenerated), an example bullet, and a note in the wrapper table's parser row: a packed numeric load is an exception like BPE ids.docs/performance.md2.18: what the Elixir side tried, the cost of the count pass, and thewrap64count that vectorises.Results
String.to_float; with an iolist it was 1.7-1.9x slower.Enum.reduce_whileper field. It was 15% slower.How it was checked
mix cigreen in the worktree on the Mac (2305 passed).test/scoreboard_doc_test.exschecks the generated tables.Agreement: every Elixir side, the kernels and their Elixir mode give the same bytes on 26 CSV and 28 JSON inputs. 411 edge fields are bit-exact with
String.to_float/1. 23 bad CSV and 26 bad JSON inputs raise on every side that has the grammar.mix defk.examples --stalelists the new row as current.Touches a promised surface (kernel subset, runtime contract, public API, error codes)? Then a
CHANGELOG.mdentry, and a deprecation for a rename or removal (docs/guides/stability.md).Touches
test/stability/? A frozen file is never edited; a release adds a directory (docs/agents/stability.md).