Chaining SelectiveTextConverter stages with TokenSelectionStrategy and preserve_tokens=True ends up converting text that was never selected.
convert_tokens_async always strips the ⟪⟫ markers, so the token branch in convert_async checks self._start_token not in result.output_text and wraps the whole output in one pair. The next stage then treats the entire prompt as selected:
stage 1 (base64, second half): tell me how ⟪dG8=⟫ ⟪ZG8=⟫ ⟪aXQ=⟫
stage 2 (rot13, token strategy): ⟪tell me how qT8= MT8= nKD=⟫
stage 3 (base64, token strategy): ⟪dGVsbCBtZSBob3cgcVQ4PSBNVDg9IG5LRD0=⟫
"tell me how" was never selected but gets base64-encoded in stage 3. This is the chaining pattern the TokenSelectionStrategy docstring describes, and the GUI stage editor from #2729 makes it easy to hit.
Fix: let convert_tokens_async keep the markers around each converted region, and have the selective converter use that when preserve_tokens is set.
Chaining
SelectiveTextConverterstages withTokenSelectionStrategyandpreserve_tokens=Trueends up converting text that was never selected.convert_tokens_asyncalways strips the ⟪⟫ markers, so the token branch inconvert_asyncchecksself._start_token not in result.output_textand wraps the whole output in one pair. The next stage then treats the entire prompt as selected:"tell me how" was never selected but gets base64-encoded in stage 3. This is the chaining pattern the
TokenSelectionStrategydocstring describes, and the GUI stage editor from #2729 makes it easy to hit.Fix: let
convert_tokens_asynckeep the markers around each converted region, and have the selective converter use that whenpreserve_tokensis set.