defk-9p6.36: codecs header measured again; range facts made base64 and hex 2x faster

docs/performance.md

@@ -203,7 +203,8 @@ Elixir job can use every scheduler.
   layout alone moves a kernel about ±3% (count_primes yielding 177 vs 180-182
   ms, with `count_primes_via_call` moving the other way). Verdicts near a
   line flip run to run: `poor_substring_search` 1.03-1.11x read 1.0x on the
-  scoreboard; `codecs` hex 2.02-2.04x reads marginal on a slow run;
+  scoreboard; `codecs` hex 2.02-2.04x read marginal on a slow run (3.6x
+  since `defk-upc`, 6.2, so no longer near the line);
   `gorilla_encode` 1.96-2.38x was marginal in 1 run of 8.
 - **Conditions:** loaded runs (load average 5-90 across sessions).
 - **Source:** `defk-wnu` (open), `defk-9p6.24` (open: proposes a ±5% tie band), `defk-9p6.13`/`.15`/`.16` comments.
@@ -788,8 +789,8 @@ For GB/s validation wrap simdutf8; a scalar kernel cannot beat the BIF by much.
 - **Numbers:** research, 16 MiB random: `:binary.encode_hex` 1,091 MB/s,
   `Base.encode16` 475 MB/s, `Base.encode64` 764 MB/s, `Base.decode64!` 551
   MB/s. `codecs`: `:binary.encode_hex/decode_hex` beat `Base` both ways; a hex
-  encode kernel is only 1.5x the BIF (now ~2.6x after `defk-upc`, under
-  load); `Base` encodes base64 faster than `:base64`, and `:base64` decodes
+  encode kernel was only 1.5x the BIF, and is 2.6-2.7x since `defk-upc`
+  (decode 4.3-4.7x; 2026-10-20, provisional); `Base` encodes base64 faster than `:base64`, and `:base64` decodes
   about 4% faster.
 - **Source:** [`string-workloads.md`](https://github.com/DanielLuning/defk/blob/main/docs/research/string-workloads.md), [`codecs.exs`](https://github.com/DanielLuning/defk/blob/main/examples/codecs.exs) (e81ee3f), `defk-9p6.19`, `defk-upc` comment 2026-10-10.
 - **Status:** research provisional; codecs measured.
@@ -1155,7 +1156,9 @@ If you must pack, see 2.2.
     it was 4.2-4.6x at 4 bytes.
   - `geohash` one point per call 0.77-0.85x (10k calls).
   - `codecs` base58: one 32-byte key per call 1.04 us against Elixir 0.94
-    us; 100 keys 2.5x; 10k keys 2.59-2.68x.
+    us; 100 keys 2.5x; 10k keys 2.59-2.68x. On 2026-10-20 (provisional):
+    one key 0.91-1.06 us against 0.86-1.43 us, 100 keys 3.9-4.0x, 10k
+    keys 3.35-3.55x; one key per call is still about level.
   - `hash_chain` HMAC: 1 nonce 0.5x, 100 nonces 0.86-0.91x, 10k 1.1-1.2x, 1M
     1.33-1.47x.
   - `utf8_valid`: at 1 KB the kernel's fixed cost (tables built per call)
@@ -1393,6 +1396,13 @@ names each kept bounds check and the fact that would drop it. See
 - **Numbers:** cosine top-k one call, 100k x 384: literal `@dim 384` 20 ms,
   `dim = length(q)` 25 ms (every access kept its check). Range facts for `2 * i`
   (`defk-upc`): 1 MB hex encode 0.601 -> 0.333 ms, decode 0.644 -> 0.503 ms.
+  The same facts for `3 * g` and `4 * g` made base64 `encode64` on 1 MB
+  489 -> 251-260 us (best of 41, three rounds, bisected over the merges of
+  `main` from 7365e4a5 to 25954894: the first fast commit is the `defk-upc`
+  merge 1ba18380, 2026-10-10). The whole codecs base64 round trip went from
+  1.04 ms (2.9-3.0x) in the example's first header to 0.47-0.50 ms
+  (6.1-6.7x) on 2026-10-20 (`defk-9p6.36`, provisional: other processes
+  used 0.6-2.3 cores).
   `pcm_mix`: a variable block offset keeps two bounds checks in the sum loop,
   which then runs scalar and takes ~70% of the kernel (0.2-0.29 of 0.3-0.37
   ms); the check-free clamp loop beside it vectorises at width 8. Since
@@ -2699,7 +2709,8 @@ The wrapper jobs and what would reopen each are in
     against 12.6-12.7 ms for its inner-product index on pre-normalised
     vectors.
   - Level: siphash's C NIF and the kernel batch on 1k keys (0.96-1.0x); the
-    fast64 C NIF is 2.1x slower than the base64 kernel.
+    fast64 C NIF is 4.1-4.4x slower than the base64 kernel (2.1x before
+    `defk-upc`, 6.2).
 - **Conditions:** pcm_mix, ws_masking, batched_levenshtein, hll, gorilla,
   cosine under load; hashes and codecs load not recorded.
 - **Source:** [`pcm_mix.exs`](https://github.com/DanielLuning/defk/blob/main/examples/pcm_mix.exs), [`ws_masking.exs`](https://github.com/DanielLuning/defk/blob/main/examples/ws_masking.exs), [`batched_levenshtein.exs`](https://github.com/DanielLuning/defk/blob/main/examples/batched_levenshtein.exs), [`hll_update_many.exs`](https://github.com/DanielLuning/defk/blob/main/examples/hll_update_many.exs), [`gorilla_encode.exs`](https://github.com/DanielLuning/defk/blob/main/examples/gorilla_encode.exs), [`cosine_topk.exs`](https://github.com/DanielLuning/defk/blob/main/examples/cosine_topk.exs), [`hashes.exs`](https://github.com/DanielLuning/defk/blob/main/examples/hashes.exs), [`codecs.exs`](https://github.com/DanielLuning/defk/blob/main/examples/codecs.exs).

examples/README.md

@@ -61,7 +61,7 @@ Measured on Apple M4 Pro (12 logical cores) in 2 runs, on 2026-10-19. The newest
 | ✅ | [`byte_histogram`](byte_histogram.exs) (measured 2026-10-19 at `8fa900b7`) | histogram of 16 MB, 12 chunks each side | **8.4×** | – | windowed entropy 11×; one process vs one call 11–18× |
 | ✅ | [`keccak`](keccak.exs) (measured 2026-10-19 at `8fa900b7`) | Keccak-256 of 10k × 32 B per call | **7.7×** | 0.63× ex_keccak | the NIF blocks its scheduler on a 16 MB message; SHA3-256 against `:crypto` is 0.81× |
 | ✅ | [`image_blur`](image_blur.exs) (measured 2026-10-19 at `8fa900b7`) | grayscale + 3×3 box blur, 1080p, 12 bands each side | **7.4×** | – | one process against one call 26–29× |
-| ✅ | [`codecs`](codecs.exs) | base64 round trip of 1 MB | **6.1×** | 4.2× fast64 | hex 4.4×, base58 3.5× per batch; one key per call is no faster |
+| ✅ | [`codecs`](codecs.exs) | base64 round trip of 1 MB | **6.1×** | 4.2× fast64 | hex 4.4×, base58 3.5× per batch; one key per call is about as fast as Elixir |
 | ✅ | [`sort_u64_keys`](sort_u64_keys.exs) | sort 1M packed keys | **6.0×** | – | 3.8× when the caller needs a list back |
 | ✅ | [`lz4_snappy_decode`](lz4_snappy_decode.exs) (measured 2026-10-19 at `8fa900b7`) | Snappy block decode of 64 KB | **5.1×** | 2.4× snappyer as installed (-O0) | slower than the same libraries built -O2; LZ4 the same picture |
 | ✅ | [`monte_carlo_rtp`](monte_carlo_rtp.exs) | 100M slot spins on all schedulers | **3.9×** | – | ≈1× at 1M spins: short kernel chunks stay on few schedulers |
@@ -178,9 +178,9 @@ has them in full.
   level at 16 MB, where it blocks its scheduler 19 ms. For SHA3-256,
   `:crypto` (OpenSSL) beats the kernel.
 - [`codecs.exs`](codecs.exs): base64, hex and base58 round trips. base64 at
-  2–2.5 GB/s, well ahead of the fast64 C NIF; hex sits at the 2× bar
-  (encode alone 1.5×: use the BIF); base58 is good on a batch of 10k keys
-  and no faster one key per call.
+  4.5–5.5 GB/s, 4× the fast64 C NIF; hex 3.6× (encode alone 2.6× the
+  BIF, decode 4.5×); base58 is good on a batch of 10k keys and about as
+  fast as Elixir one key per call.
 - [`sort_u64_keys.exs`](sort_u64_keys.exs): heapsort kernel on 1M packed
   keys against decode plus `:lists.sort`; still good when the caller needs a
   list back, with the decode counted.

examples/_tables.exs

@@ -56,7 +56,7 @@
     "codecs" => %{
       job: "base64 round trip of 1 MB",
       native: "{native} fast64",
-      line: "hex {speedup:codecs_hex}, base58 {speedup:codecs_base58} per batch; one key per call is no faster"
+      line: "hex {speedup:codecs_hex}, base58 {speedup:codecs_base58} per batch; one key per call is about as fast as Elixir"
     },
     "cosine_topk" => %{
       job: "top-10 of 100k × 384 f32 embeddings",

examples/codecs.exs

@@ -16,8 +16,8 @@
 #
 # Pre-registered (ticket): base64 good only if a table-driven kernel reaches a
 # few GB/s, hex marginal (the BEAM's hex is a C BIF), base58 good.
-# Expected verdicts, as measured: base64 good, hex good (at the 2x line,
-# differs from the ticket), base58 good.
+# Expected verdicts, as measured: base64 good, hex good (3.6x, differs from
+# the ticket), base58 good.
 #
 # Kernels: [u8] in. encode64 and encode16 return [u8] of the exact length
 # (4 * ceil(n / 3), 2n); decode64 returns {buffer, length} (3 per quad less
@@ -53,8 +53,8 @@
 #   base58  b58 1.0.3 and erl_base58 0.0.1 from Hex (bignum, digit by digit),
 #           and Codecs.Elixir below: one bignum divided by 58^9 per step
 #           (nine digits from a small integer) and decoded nine characters
-#           per step: encode 3.8x faster than b58 and 7x than erl_base58,
-#           decode 9.7x and 1.6x. Both packages raise ArithmeticError on a
+#           per step: encode 5.4-5.6x faster than b58 and 10.4-10.8x than
+#           erl_base58, decode 9.7-10.8x and 3.0-3.1x. Both packages raise ArithmeticError on a
 #           bad character, not ArgumentError, and erl_base58 decodes "1" to
 #           two bytes; neither is in the bad-input check.
 #   Not tried: a batch Elixir base58 (no shared work between keys to save).
@@ -76,33 +76,40 @@
 # Codecs.Kernels.__defk_elixir__/2) on the small inputs and every
 # bad input.
 #
-# Measured (M-series Mac, OTP 29, Elixir 1.20, best of 15; three runs):
+# Measured (M-series Mac, OTP 29, Elixir 1.20, best of 15; three runs on
+# 2026-10-20, provisional: other processes used 0.6-2.3 cores beside them):
 #
-#   base64, 1 MB    Elixir 3.08-3.17 ms  defk 1.04-1.05 ms  2.94-3.04x  good
-#                   (encode 0.49 ms = 2.1 GB/s, decode 0.56 ms = 2.5 GB/s of
-#                   input); fast64 2.19-2.22 ms, so the kernel is 2.1x
-#                   faster than the anchor NIF. Sweep 64 KB 2.6-2.8x, 16 MB
-#                   3.1-3.4x. Ticker gap while encoding 64 MB on one
-#                   scheduler: Elixir 8.8-10.1 ms, defk 4.6-4.8 ms, fast64
-#                   4.4-5.0 ms (defk yields every 250 us in its loop; the gap
-#                   left is the 85 MB output alloc before the loop: a kernel
-#                   that only allocs 85 MB gives a 5.0 ms gap).
-#   hex, 1 MB       Elixir 2.55-2.59 ms  defk 1.25-1.28 ms  2.02-2.04x  good
-#                   (encode 1.5x over :binary.encode_hex, decode 2.5x over
-#                   :binary.decode_hex); sweep 1.74-2.05x. At the 2x line:
-#                   the 5% tie band (good from 1.9x) keeps a slower
-#                   run good, and for encoding alone the BIF is the
-#                   recommendation.
-#   base58, 10k     Elixir 12.7-13.2 ms  defk 4.8-5.1 ms    2.59-2.68x  good
-#   keys per call   (kernel encode 1.9 ms, decode 2.3 ms). One key per call:
-#                   Elixir 0.94 us, defk 1.04 us per round trip, so a
-#                   single-key call is no faster than Elixir (sweep: 1 key
-#                   per call 1.13x, 100 keys 2.5x, 10k keys 2.3-2.9x).
+#   base64, 1 MB    Elixir 3.05-3.11 ms  defk 0.47-0.50 ms  6.1-6.7x    good
+#                   (encode 0.23 ms = 4.6 GB/s, decode 0.24-0.25 ms = 5.5-5.8
+#                   GB/s of input); fast64 2.03-2.04 ms, so the kernel is
+#                   4.1-4.4x faster than the anchor NIF. Sweep 64 KB
+#                   5.0-5.2x, 16 MB 6.4-6.6x. Ticker gap while encoding 64 MB
+#                   on one scheduler: Elixir 6.9-9.3 ms, defk 2.15-2.23 ms,
+#                   fast64 4.2-5.1 ms (defk yields in its loop and zeroes the
+#                   85 MB output in strips; a 1 ms ticker reads about 2 ms
+#                   under load on every side).
+#   hex, 1 MB       Elixir 2.38-2.43 ms  defk 0.67 ms       3.56-3.63x  good
+#                   (encode 2.6-2.7x over :binary.encode_hex, decode 4.3-4.7x
+#                   over :binary.decode_hex); sweep 64 KB 2.9-3.3x, 16 MB
+#                   3.3-4.0x.
+#   base58, 10k     Elixir 11.2-11.8 ms  defk 3.28-3.35 ms  3.35-3.55x  good
+#   keys per call   (kernel encode 1.7 ms, decode 1.5-1.6 ms). One key per
+#                   call: Elixir 0.86-1.43 us, defk 0.91-1.06 us per round
+#                   trip (sweep: 1 key per call 1.3-1.4x, 100 keys 3.9-4.0x,
+#                   10k keys 2.5-4.1x).
 #
-# Recommendation: base64 over a few KB and batched base58 are kernel jobs
-# (and need no hand-written chunking); for hex, :binary.encode_hex and
-# :binary.decode_hex are within 2x and already native; for one base58 key at
-# a time, the Elixir function below is as fast.
+# The kernels were 2-2.5x slower when this block was first written
+# (2026-10-09: base64 1.04 ms, hex 1.25 ms, base58 4.8-5.1 ms). For base64
+# and hex the change is the compiler's range facts for constant multiples
+# of a loop index: the loops index s[3 * g] and out[4 * g] (hex: 2 * i),
+# and those bounds and overflow checks went (encode64 alone 489 -> 251-260
+# us, bisected to that change). The base58 gain was not bisected.
+#
+# Recommendation: base64 over a few KB, hex over a few KB and batched
+# base58 are kernel jobs (and need no hand-written chunking); for hex the
+# BIFs are already native and need no C compiler, so they stay the choice
+# where 2.6x does not matter; for one base58 key at a time, the Elixir
+# function below is about as fast.
 
 Mix.install([
   {:defk, path: Path.expand("..", __DIR__)},
@@ -880,7 +887,7 @@ _r64 =
       native: fn -> Fast64.encode64(big) end
     ],
     reason:
-      "a table-driven kernel reaches 2-2.5 GB/s, 3x Base and 2x the fast64 NIF, with yielding instead of hand-written chunking"
+      "a table-driven kernel reaches 4.5-5.5 GB/s, 6x Base and 4x the fast64 NIF, with yielding instead of hand-written chunking"
   )
 
 sweep16 = fn size ->
@@ -904,7 +911,7 @@ r16 =
     sweep: {[64 * 1024, mb, 16 * mb], sweep16},
     sweep_runs: 5,
     reason:
-      "at the 2x line: decode 2.5x over :binary.decode_hex, encode 1.5x over :binary.encode_hex, which is a C BIF already"
+      "decode 4.5x over :binary.decode_hex and encode 2.6x over :binary.encode_hex, though both are C BIFs already"
   )
 
 per_call = fn n ->
@@ -937,7 +944,7 @@ r58 =
       end,
     sweep_runs: 5,
     reason:
-      "bignum base58 in Elixir against limb arithmetic in a kernel, batched; one key per call is no faster than Elixir"
+      "bignum base58 in Elixir against limb arithmetic in a kernel, batched; one key per call is about as fast as Elixir"
   )
 
 for r <- [r16, r58], not r.correct or r.verdict != r.expected do