feat(qwen4_exp): image input (vision tower + mRoPE) for Qwen3.8-Flash-Next

[gdevenyi]

What this adds

Image input for qwen4_exp (Qwen3.8-Flash-Next) through /v1/chat/completions. FreeToken serves this model text-only today: the loader drops the 333 model.visual.* tensors and render_messages refuses every non-text part (#321, #348). Opt-in with FREETOKEN_LOAD_VISION=1 (the tower is 0.86 GiB of bf16 per rank). Video is not covered.

Stacked on #385 (the tensor-parallel PR): this branch contains that commit because the loader's _shard and the attention layer's TP-aware forward are the seams it hooks into. Review that one first; the vision commit is the last of the three here (the middle one is #385's loader fix).

How it works

• Vision tower (models/qwen4_exp/vision.py). The HF Qwen4ExpVisionModel (27 blocks) runs inside a BaseOP whose tensors travel as visual.* in the model state dict, so the engine loads them with the dense weights: the expert-cache planner counts them, --dummy-weight fills them, and every TP rank holds the same copy. The op is built on the meta device like the rest of the model; load_state_dict assigns the loaded tensors in place (no second GPU copy) and rebuilds the non-persistent rotary buffer on the device.
• mRoPE (models/qwen4_exp/mrope.py). Qwen3.8 ropes image tokens on three axes (T/H/W, interleaved sections 11/11/10) and text after an image continues from max(position) + 1. rope_index ports HF get_rope_index; mrope_cos_sin builds the interleaved cos|sin rows. A prefill batch with image tokens gets a per-token table [T, rotary_dim] in the same layout as RotaryEmbedding._cos_sin_cache, and the existing flashinfer / triton rope kernels index it with row numbers (Batch.rope_positions); decode reads the normal cache at position + delta. The QSA indexer ropes each compressed key at its group's first token, so the table carries index_ratio - 1 lead rows per request for groups that straddle a chunk boundary. No new kernels. Text-only batches alias positions and run exactly the kernels they ran before.
• Request path. image_url parts (inline data: URLs only, 16 MiB cap; remote URLs are refused rather than fetched on the client's behalf) are decoded in the API server and kept as {"type": "image"} parts for the chat template. The tokenizer worker runs the checkpoint's image processor (FREETOKEN_IMAGE_MAX_PIXELS, default 1280*28*28, about 1,000 tokens per image) and expands each <|image_pad|> to its soft-token count; the scheduler encodes the images on its rank and computes the rope positions before admission. Image prompts chunk like text prompts: each chunk scatters only the soft-token rows whose placeholders fall inside it (the old path raised NotImplementedError inside the scheduler). The wire encoder now carries N-D tensors (it asserted 1-D).

Other adapters (Anthropic, Responses) keep refusing image parts with the existing error.

Status

Verified on the real checkpoint on CPU (tower load through the state-dict contract, rope positions and cos|sin rows equal to HF, tokenizer path, weight iteration, wire round trip, 60 tests in tests/models/qwen4_exp plus the scheduler / tokenizer / server suites) and end to end on 2 x RTX 6000 Ada at TP=2: see the results comment below (image answers correct for one and two images and for a text follow-up; the loaded tower costs 2.6 points of expert residency and no decode or TTFT). The three GPU test failures on this box are pre-existing on main (see the comments). Draft only because of the open-PR cap; ready for review from my side.

• Prefix cache for image prompts (ed0982a). Image placeholders share one token id, so multimodal requests used to be kept out of the shared prefix cache and every turn of a conversation holding an image re-prefilled the whole context (measured before the change: 20 turns of 109k-122k tokens, #cached-token: 0 each, ~35 s per turn at TP=2). The tokenizer worker now emits cache_ids next to the expanded input_ids: the same tokens with each image's placeholder run replaced by ids derived from a blake2b hash of the image bytes (>= 2**30, above any vocabulary, hash + offset within the run). The cache manager keys match/insert on cache_ids when present, else on input_ids; the model still reads input_ids, and the per-chunk placeholder window already skips cached placeholders. Verified on the TP=2 server: turn 2 after an image hits (cached=320 of a 330-token prompt), an identical request hits fully, the same text with a different image matches nothing past the text (answer describes the new image), and a 26k-token prompt with the image after a 20k-token preamble prefills in 4 chunks then hits cached=26304 on turn 2. tests/scheduler/test_mm_cache_key.py, test_mrope.py::test_image_cache_ids.

Testing

• tests/models/qwen4_exp/test_mrope.py (CPU; the HF comparisons skip without a transformers that ships qwen4_exp).
• tests/models/qwen4_exp/test_mrope_gpu.py (GPU).

🤖 Generated with Claude Code

https://claude.ai/code/session_0173pf9k9fSVtwbm3f898HDt

[gdevenyi]

GPU results, 2 x RTX 6000 Ada at TP=2 (#385), RadixArk NVFP4, --num-tokens 262144 --memory-ratio 0.94 --max-running-requests 16, this branch with FREETOKEN_LOAD_VISION=1:

Image probe (rendered 640x480 PNGs as data: URLs, temperature 0, thinking off):

case	prompt tokens	answer
one image: "HELLO 42" over a blue circle, "what text / what colour"	332	HELLO 42
text follow-up in the same conversation: "spell it backwards"	357	24 OLLEH
two images in one message (second: "GREEN TEA" over a green circle)	635	first: "HELLO 42", solid blue circle; second: "GREEN TEA", solid green circle
text-only request	21	a normal one-sentence answer

The follow-up exercises decode after an image (rope at position + delta) and a second prefill that carries the image history; the two-image case exercises several images per prompt in order.

Cost of the loaded tower, same flags, same tree with vision off vs on:

	single-stream tok/s	8 concurrent tok/s	expert residency	TTFT (1k)	expert slots
vision off	88.9	325.5	94.5%	0.84 s	23,229
vision on	89.9	325.8	91.9%	0.83 s	22,594

So the 0.86 GiB tower per rank costs 2.6 points of expert residency and nothing else here. Greedy 256-token continuations of three text prompts: the code prompt and the ~1k-token prompt are word-for-word identical between the two runs (and to production without this branch); the short essay prompt diverges after 84 words, which is the same short-prompt run-to-run noise seen between two passes of the same TP=2 server (bf16 atomics in the expert kernels), not a rope difference: the 1k prompt is the one that exercises the QSA blocks.

tests/models/qwen4_exp on CPU: 60 passed. The GPU pass of the same package in the chain reported 3 failures whose names the harness did not keep; I am rerunning it with full output and will attribute them.

[gdevenyi]

GPU test pass of tests/models/qwen4_exp on one RTX 6000 Ada, same box: this branch 107 passed / 3 failed, #385 alone 97 / 3, plain main (af71ba4) 94 / 3. The three failures are the same on all three trees: test_qsa_backend.py::test_chunked_prefill_matches_one_shot[unaligned|page-boundary|boundary+1], whose torch.equal between chunked and one-shot prefill is off by bf16 noise here (torch 2.11.0+cu130, flashinfer 0.6.18, triton 3.6.0, sm_89). Pre-existing, unrelated to this PR. The new test_mrope_gpu.py (table path == cache path through the QSA layer, chunked continuation and decode) passes.

[gdevenyi]

Pushed 28fd56d: chunked prefill for image prompts. The first version refused image prompts longer than one prefill chunk (--max-extend-tokens, 8192 by default); a coding-agent client with a ~109k-token context hit that at once. prefill.py now chunks them like text and the scheduler scatters, per chunk, only the soft-token rows whose placeholders fall inside that chunk (_mm_embeds_window, unit test in tests/scheduler/test_mm_window.py); the mRoPE table already windowed per chunk. CPU packages on the box: 713 passed, 100 skipped. The GPU end-to-end run with a 20k-token preamble before the image is queued for the next restart window; I will post it here.

[gdevenyi]

GPU end-to-end for the chunked case (TP=2, production flags, 2026-09-04 18:45): a 23,504-token prompt (20k-token text preamble, then the image, then the question) prefilled in three chunks (8192 / 8192 / 7120) and answered Text: HELLO 42 / Circle colour: blue in 6.2 s; the one-, two-image and text-only cases are unchanged. The client whose 109k-token context was refused before goes through the same path.

[MT-z]

Ran this branch at TP=1 on one RTX 4090, 24 GiB, in a 61 GiB box. It
works, including the chunked path you added last night. The only thing standing between it
and a machine this size is that the model needs a second PR to fit in host RAM at all.

Both image cases answered correctly at temperature 0, FREETOKEN_LOAD_VISION=1:

case	prompt tokens	prefill chunks	answer
one 448x448 PNG, "list every shape with its colour"	263	1	Red triangle, blue rectangle, green circle
~20k words of filler, then the same image, then the question	22,439	4096 x 5 + 1959	Red triangle, blue rectangle, green circle.

The second is the case 28fd56d fixes. The same request against 70d9e8d came back
image prompts must fit in one prefill chunk: 22439 tokens > 4096 (--max-extend-tokens).
Here --max-extend-tokens is 4096, so the image lands in the sixth chunk rather than the
first, and the shapes still come out right.

Getting there needs #337 as well, on a box this size. Without the disk tier the expert
bank build reaches the cgroup ceiling and is OOM-killed in about half a minute, at 46 GiB
and again at 50 GiB; the checkpoint carries 73.3 GiB of expert tensors, which is simply
more than this machine has. With --moe-disk-tier on --expert-ram-experts 224 (224 of 512
per layer resident, the rest fetched from the checkpoint) the same tree peaks at 43 GiB and
serves. So the two PRs are a pair here rather than alternatives -- which also puts the
requirement at a 24 GB card and about 64 GB of system RAM. If anyone else has that, the
flags at the bottom are the whole configuration.

They merge almost cleanly, so the recipe is short. One conflict, models/nvfp4_banks.py,
two blocks: #337 adds rows_per_layer to bound how many expert rows the loader
materializes, #385 replaces the loose bank variables with the TP-aware _Placer, and the
two are orthogonal -- keeping both lines resolves it. The function bodies auto-merge into
the right hybrid on their own (place.put(...) for the writes, rows_per_layer for the
tracker and the skip), so there is nothing else to hand-resolve.

Tests on the merged tree, against plain main (af71ba4) on the same box:

                merged            main
tests/moe       127 passed        (disk-tier tests are #337's)
tests/models    4 failed 165 p    4 failed 151 p
tests/kernels   3 failed 214 p    3 failed 214 p
tests/scheduler 88 passed         88 passed
tests/engine    108 passed        108 passed

Same seven failures either way -- the test_qsa_backend.py::test_chunked_prefill_matches_one_shot
family you already attributed, plus four in tests/models that are on main too.

Measured on an RTX 4090 (24 GiB, sm_89) / i9-14900KF / 61 GiB box, driver 595.84, CUDA
13.3, torch 2.11.0+cu130, triton 3.6.0, sgl_kernel 0.4.5, freetoken 0.1.2, model
RadixArk/Qwen3.8-Flash-Next-NVFP4, TP=1. The tree is main af71ba4 + #337 @ a6bd5c0 +
this branch @ ed0982a. Server flags:

FREETOKEN_LOAD_VISION=1 ft serve --model-path RadixArk/Qwen3.8-Flash-Next-NVFP4 \
  --moe-disk-tier on --expert-ram-experts 224 --disable-moe-prefill-overlap \
  --cuda-graph-max-bs 0 --moe-cache-auto --max-running-requests 2 \
  --memory-ratio 0.90 --kv-reserve-tokens 32768 --max-prefill-length 4096

Run inside a systemd scope with MemoryMax=46G and MemorySwapMax=0, which is where the
OOM figures above come from.

The only edit anywhere was that one merge conflict in #337's nvfp4_banks.py; nothing in
this branch itself was changed. This box stays available if there is anything you want run
at TP=1 on a single 24 GiB card.

Assisted-by: Claude Opus 5

[smilegiromg-sudo]

Verified on Ubuntu 26.04 — image input works, with two notes

Environment: Ubuntu 26.04, Python 3.14 (editable install from source), torch 2.11.0+cu130,
torchvision 0.26.0+cu130, single GPU, RadixArk/Qwen3.8-Flash-Next-NVFP4,
FREETOKEN_LOAD_VISION=1.

Image input works: a 256x256 solid-red PNG answers 红, a solid-green one 绿,
prompt_tokens=84 (64 image soft tokens + template/text), finish_reason=stop. Multi-turn text
conversations and prefix caching keep working (#cached-token 192/1408 observed), decode ~55-70
tok/s single stream. The chunked-prefill and cache-key commits both behave as described.

1. Two undeclared dependencies

1. pillow — _decode_images() does from PIL import Image, but pillow is not in
   pyproject.toml. transformers only pulls it via its [vision] extra, so a base install fails
   with ModuleNotFoundError: No module named 'PIL'.
2. torchvision — AutoImageProcessor.from_pretrained() requires it; a base install fails with
   AutoImageProcessor requires the Torchvision library but it was not found.

Pin torchvision to the torch version when installing: on the cu130 index the latest is 0.29, which
depends on torch==2.14.0 and would silently upgrade torch past this project's
torch>=2.11,<2.12 constraint. pip install "torchvision==0.26.*" --index-url https://download.pytorch.org/whl/cu130 works.

Also, the from PIL import Image in _decode_images() sits outside the try, so a missing
dependency escapes as the generic tokenization failed for request N: .... Detecting it in
_vision_processor() and raising something like
Pillow is required for image input: pip install pillow would have saved me two restart cycles.

2. The prefix-cache key only carries 30 bits of the 64-bit hash

In _image_cache_ids():

mask = _IMAGE_KEY_BASE - 1                                  # 2**30 - 1
run = (torch.arange(n, dtype=torch.int64) + (h & mask)) & mask

blake2b(digest_size=8) gives 64 bits, but & mask keeps only the low 30 — about 1.07e9 keys.
By the birthday bound that's a ~5% chance of a collision after 10k distinct images and ~68% after
50k. A collision makes two images' placeholder runs byte-identical, so the radix tree hits and
serves the other image's KV — a silently wrong answer, not a crash. (The wrap-around in
(h + j) & mask also lets runs overlap partially, which is harder to notice.)

The id field can't simply be widened: input_ids is int32 and base + run already tops out at
2**31-1.

One fix that needs no tree changes — spread the whole hash across the run's n placeholders:

# low 29 bits: (hash + offset), as today; bit 29: one more hash bit, a different one per position
run = torch.tensor([
    (((h & low_mask) + j) & low_mask)
    | (((h >> (SPAN_BITS + (j % HIGH_BITS))) & 1) << SPAN_BITS)
    for j in range(n)
], dtype=torch.int64)

With SPAN_BITS = 29 and HIGH_BITS = 64 - 29 = 35, a run of n >= 35 placeholders — real images
are hundreds — carries every bit of the hash, so two runs match only when the full 64-bit hash
does: collision odds drop from 2**-30 to 2**-64. Runs shorter than 35 degrade toward the old odds.

I have this plus the pillow/torchvision handling running locally (14 unit tests, including one that
flips each of the 64 hash bits and asserts the key changes); happy to open it as a PR against your
branch if useful.

Thanks for the PR — the mRoPE port and the state-dict contract for the tower are clean, and the
pre-existing-failure attribution in the description made review much easier.

(Review text drafted with AI assistance; all findings reproduced on real hardware.)

[lukascechovic]

Read this PR alongside #385 because we shipped the same two features independently and have been
serving them for a week — vision and TP=2 for this model on two AMD R9700s (gfx1201, 32 GiB
each). Most of it matches what we ended up with, including things we expected to disagree about.
Two findings are worth more than that agreement, because both are things our hardware made us hit
and yours probably will not, until it does.

Everything below is full-box on 2 × R9700 unless said otherwise, measured on our own tree
(upstream 4b94bdc3 plus a local patch series), not on main. No quality claim — we have no
fidelity instrument on this box.

---

1. _prepare_multimodal's except is a per-rank early return inside a collective region

In scheduler/scheduler.py:

if msg.mm_inputs is not None:
    error = self._prepare_multimodal(msg)
    if error is not None:
        logger.warning_rank0(f"Rejecting request {msg.uid}: {error}")
        self.send_result([ErrorReplyMsg(uid=msg.uid, error=error)])
        return                      # <-- skips prefill_manager.add_one_req(msg)
self.prefill_manager.add_one_req(msg)

and inside it:

except Exception as exc:  # noqa: BLE001 -- a bad image must not take the scheduler down
    logger.warning_rank0(f"image encoding failed for request {msg.uid}: {exc!r}")
    return f"could not encode images: {exc}"

The docstring says "every TP rank sees the same UserMsg", and that is true of the input. It
is not true of the outcome. At this PR's head (ed0982ae) the helper has exactly two
early-return paths, and only one of them is deterministic in the request: no vision tower, which
every rank decides the same way. The other is the bare except, and the exception that matters
there is torch.OutOfMemoryError — which depends on that rank's card at that instant. When it
fires on one rank only, that rank returns and never enters the forward pass while its peer does —
and the two ranks are now in different places in the collective schedule.

28fd56d made that surface larger, not smaller. The one-chunk guard it removed
(len(msg.input_ids) > self.prefill_budget) was deterministic in the request, so it was the safe
kind of early return — taking it out leaves the per-rank kind as one path in two rather than one in
three. Chunked prefill is clearly the right change; this is just where its blast radius went.

The comment is right that a bad image must not take the scheduler down. What actually happens is
that the process comes down anyway, about two minutes later, for a different reason. This is our
log, from the deployed TP=2 vision row, driving it by hand from a chat UI:

13:38:32  rank 0  WARNING  vision encode failed for request 12: OutOfMemoryError(
                           'CUDA out of memory. Tried to allocate 2.21 GiB. GPU 0 has a total
                            capacity of 31.86 GiB of which 476.00 MiB is free. ...')
13:38:32  rank 0  Scheduler is idle, waiting for new reqs...
                  ⇒ rank 1 is still inside ALLREDUCE SeqNum=3359
13:39:43  rank 1  Watchdog caught collective operation timeout: WorkNCCL(SeqNum=3359,
                  OpType=ALLREDUCE, NumelIn=6479360, Timeout(ms)=60000) ran for 60008 ms
13:40:43  rank 1  "To avoid data inconsistency, we are taking the entire process down."
13:40:49  FrontendAPI  ERROR  Backend supervisor: backend worker exited
13:40:59  FrontendAPI  ERROR  Backend worker is gone and cannot be restarted; stopping the API server

Two properties make this nastier than the timestamps suggest:

• The client sees no error. The request that triggers it is refused cleanly, and the next
  request just hangs until the server disappears. Nothing in the response says an input was
  dropped or a rank died — which is the same complaint as The two protocol surfaces disagree on image input: /v1/chat/completions 400s, /v1/messages silently drops the image and answers #348, arriving by a different route.
• The cause has scrolled away by the time you look. The OOM is at 13:38:32; the visible failure
  is at 13:40:59. Our engine log ring holds ~14 minutes, so we got lucky. A shorter ring, or a
  restart policy that recycles the worker, and all you have is "backend worker is gone".

This is not --disable-pynccl failing to apply, and it is not specific to ROCm. That flag
disables the pynccl fast path; the fallback is torch's own ProcessGroupNCCL, which is what timed
out — working exactly as configured. The mechanism is the control-flow divergence, and it needs
only two ranks and one per-device condition.

Our trigger was ours, and you do not share it. The 2.21 GiB in that log is our own port
padding the pixel batch to n_images × p_max; your tokenizer ships the processor's flat
pixel_values straight through and never asks for that allocation. So do not read the log as a
prediction about your box — read it as a demonstration that any per-rank OOM, from any source,
ends this way at TP>1. The handler is what makes the source not matter.

It is also timing-sensitive in a way that hides it. distributed_timeout in
engine/config.py is 60.0 with no CLI flag, so the desync becomes a kill after 60 s. #385 raises
that default to 1800 for a good and unrelated reason (ranks reaching their first collective minutes
apart behind a large load) — and one side effect is that this failure would take thirty minutes
to surface instead of two, as an unexplained hang.

What we did about it

The fix upstream's own source already names. scheduler/scheduler.py carries a comment about
deferred "all-rank failure-agreement machinery"; that is exactly what is missing. We added a
patch that all-reduces the local failure flag over the CPU (gloo) process group and makes every
rank act on the agreed outcome:

def any_rank_failed(local_failed: bool, tp_size: int, reduce_max: ReduceMax) -> bool:
    if tp_size <= 1:
        return local_failed
    flag = torch.tensor([1 if local_failed else 0], dtype=torch.int64, device="cpu")
    reduce_max(flag)
    return bool(int(flag.item()) != 0)

so a one-rank encode failure becomes a refused request on a live row, which is what the
except was trying to achieve. Three notes if you take this direction:

1. Use the CPU group, not the device group. The agreement must not itself enter the NCCL stream
   the desync has already corrupted. In this engine tp_cpu_group is gloo in both branches of
   engine._init_communication, not only under --disable-pynccl.
2. Give that group an explicit timeout. A bare new_group(backend="gloo") inherits torch's
   default_pg_timeout — 30 minutes on our pinned torch — so the agreement collective can outlive
   the thing it is meant to bound. We pass 60 s explicitly.
3. sync_all_ranks() in server/launch.py is not the primitive. It is a barrier; it makes the
   ranks meet, but it does not carry which outcome they agreed on.

We are happy to open this as a PR against #385 or #386 if that is useful — say which and we will
rebase it onto whichever lands first.

One thing we have not established, and will not claim: this chain is a race. We have one
organic occurrence, read from logs, and a deliberate reproducer we have not run. A test run that
does not produce the desync shows only that it did not fire, never that it cannot.

---

2. FREETOKEN_IMAGE_MAX_PIXELS bounds one image. Nothing bounds how many arrive

tokenize_one hands the processor's pixel_values through in its native flat layout, so the host
cost is linear in the total patch count — and nothing in the request path caps n_images. The
env var bounds each image; a request carrying a few hundred small ones is legal under it and is paid
for in full.

We know the shape of that cost because we measured it on our own port, which got the layout wrong
first — we padded to n_images × p_max and had to patch our way back to the packed form you already
have. Two things from those measurements survive the difference and apply here:

• The sum term is real, and a good layout does not remove it. Fitting tokenizer-worker
  residency across a five-rung ladder, the packed content itself carried a coefficient of ~0.35 —
  roughly a third of the decoded patch bytes stay resident per request on top of the copy in flight.
  On a request with a few hundred images that term alone is the whole budget, and it is the term
  your layout already reduced to its floor. There is nothing left to win there except a count bound.
• The ordering pays before it checks. tokenize_one decodes every image and runs the
  processor before anything consults a length, so a hostile or merely careless request is paid for
  in full and only then, maybe, refused. An admission bound on n_images and on Σ(patches), taken
  before the first Image.open, is cheap, needs no layout change, and is the half of this we would
  do first.

Worth a container memory cap on the tokenizer worker regardless of any of the above. When this
went wrong for us the worker took the whole box down with it; under a cap the kernel kills the worker
and the box survives. That is blast-radius control, not a fix — but it is the difference between a
row restart and a reboot.

What we are not claiming: that you have our bug. You do not. pixel_values is never padded to
p_max anywhere in this PR — we checked the whole diff — and that is precisely the thing we had to
go back and fix in ours. This section is here because n is still unbounded on the term that
remains.

---

3. Where we converged, for whatever it is worth as corroboration

We arrived at two of your design decisions independently, which is probably the most useful thing we
can say about them:

• Chunked image prefill. We hit the same NotImplementedError in the scheduler and solved it
  the same way — each chunk scatters only the soft-token rows whose placeholders fall inside it.
• An image-keyed prefix cache. We key on the picture too, from a blake2b hash of the image
  bytes, for the same reason you give: otherwise every turn of a conversation holding an image
  re-prefills the whole context. Our measurements agree with yours that this is the difference
  between #cached-token: 0 every turn and a full hit.

On the key width, since @smilegiromg-sudo's review landed a few hours before this comment and
touches the same function. Their 30-bit collision finding on _image_cache_ids() matches our
reading, and our construction keeps more of the digest, so it may be a useful second datapoint.
We replace only the first two placeholders of each image's run with negative int32 markers
carrying bits [0,31) and [31,62) of the same 8-byte blake2b digest:

_MARK_BITS = 31
_MARK_MASK = (1 << _MARK_BITS) - 1
# two negative int32 marker ids from an 8-byte digest: bits [0,31) and [31,62)
return -(h & _MARK_MASK) - 1, -((h >> _MARK_BITS) & _MARK_MASK) - 1

That is 62 bits at a run length of 2, and 31 bits on a 1-token run. The ids stay inside int32
because input_ids is int32 on the wire, and negative markers cannot collide with a vocabulary
id, so the run needs no reserved base and no wrap-around. Against @smilegiromg-sudo's
spread-the-hash fix the tradeoff is only run length: theirs reaches the full 64 bits but wants
n >= 35 placeholders, ours reaches 62 at n >= 2. Either beats 30 — we mention ours because short
runs are exactly where the masked-run form degrades and the two-marker form does not.

We recorded the residual odds rather than hedging them: a collision serves another picture's KV at
about 2^-62.

We had drafted both as upstream reports and are dropping them in favour of this comment; your
implementations landed first and cover the ground.

One measured note on the prefix-cache half, since you may hit it: skipping the encode on a cached
image is a separate win from skipping the prefill, and the tower runs at admission — before
anything consults the cache — so the ordering has to change for the encode saving to be available at
all. Also worth knowing that llama.cpp, which does skip the encode, still decodes every image
every turn (it has to, to hash it), so this is a GPU-work parity item and probably not a host-RAM
one.

---

Platform

• 2 × AMD Radeon AI PRO R9700 (gfx1201, 32 GiB each), Linux, ROCm 7.14, torch 2.11.
• qwen4_exp / Qwen3.8-Flash-Next NVFP4, --moe-backend offload, --tp-size 2, 262,144-token KV
  reserve, one running request.
• Our tree is upstream 4b94bdc3 plus a local patch series; the QSA prefill kernel runs with
  BLOCK_N clamped to 32 for RDNA's LDS budget (Qwen3.8-Flash-Next: QSA prefill kernel exceeds the 64 KiB LDS budget on RDNA, capping prompts at 15 tokens #349).
• Every number above is from our tree, not from main, and every one is full-box.
• No quality claim anywhere in this comment — we have no fidelity instrument on this box, which
  is also why we have an open question on logprobs (feat(server): OpenAI-compatible logprobs for chat and legacy completions #224).

[gdevenyi]

Rebased onto fb7f732 and re-stacked on #385. One conflict: Qwen4ExpForCausalLM.__init__, where the old lm_head_quant == "nvfp4" branch is gone (the scheme system owns lm_head quantization now) and ParallelLMHead takes quant_config / prefix. The vision tower construction is unchanged and still behind FREETOKEN_LOAD_VISION=1.

Full pytest tests with CUDA hidden: main 1205 passed / 0 failed, this branch 1226 passed / 0 failed, zero failures new relative to main. (With the GPUs visible this box is serving a model and 65-95 GPU tests fail on main itself, so only the set difference is meaningful.)

🤖 Generated with Claude Code

https://claude.ai/code/session_0173pf9k9fSVtwbm3f898HDt

[gdevenyi]

Re-tested against current main (fb7f732, i.e. after the #418 / #427 / #426 quantization refactor) on 2 x RTX 6000 Ada, TP=2 box.

Method. This PR's head merged onto main, then the full pytest tests suite. The run is CUDA-hidden (CUDA_VISIBLE_DEVICES="") on purpose: this box is serving a model on both GPUs, and with them visible 65-95 GPU tests fail on main itself with AcceleratorError: out of memory, with the count swinging ~10 between identical runs. Hiding CUDA makes the result deterministic, so a failure-set difference against main means something. Baseline: main = 1205 passed, 350 skipped, 0 failed.

Result: 1226 passed, 351 skipped, no new failures.

1 more tests skip here than on main: this PR adds that many CUDA-only tests the hidden-GPU run does not exercise. So the result above says it merges, imports and leaves every CPU-reachable path intact — it says nothing about the kernels themselves. I am running those on free cards in a maintenance window and will post the numbers here.

🤖 Generated with Claude Code

https://claude.ai/code/session_0173pf9k9fSVtwbm3f898HDt

[gdevenyi]

Status note, and an explicit statement of what tonight's testing does not cover for this PR.

This branch is in our production deployment with FREETOKEN_LOAD_VISION=1, so the vision tower is built and resident on every start. It came through a full validation run at TP=2 on 2 × RTX 6000 Ada — server start, CUDA-graph capture, prefill warmup, a three-run serving benchmark and two 300-question GSM8K passes — with no fault attributable to the tower, and with the residency cost it charges already priced in (expert residency 0.717 with the tower loaded, identical to the branch without this PR).

But every one of those requests was text. I did not run an image through it tonight. So what this buys the PR is "the tower loads, coexists with TP=2 and the NVFP4 expert path, and costs what we thought" — not "image input is correct". The mRoPE and tower-output paths still rest on the earlier GPU testing and the CPU tests in the tree.

Rebased onto fb7f732 and re-stacked on #385; the Qwen4ExpForCausalLM.__init__ conflict noted earlier is resolved and the branch merges clean onto the current deploy line.

I will post an image-path result when it next gets a window with a multimodal prompt set.

🤖 Generated with Claude Code

https://claude.ai/code/session_0173pf9k9fSVtwbm3f898HDt