Skip to content

Fix multiline interpolation and exact Rust output - #111

Merged
konard merged 20 commits into
mainfrom
issue-37-e875842b
Sep 13, 2026
Merged

Fix multiline interpolation and exact Rust output#111
konard merged 20 commits into
mainfrom
issue-37-e875842b

Conversation

@konard

@konard konard commented Sep 9, 2025

Copy link
Copy Markdown
Member

Problem

Multiline template interpolations could be treated as shell source by this
branch's special-case writer. Backticks, dollar signs, quotes, backslashes, and
newlines were therefore corrupted or executed when used in commands such as:

await $`echo "${complexContent}" > ${testFile}`;

The branch also diverged from the generic literal-interpolation design now on
main, and its fallback path broke multiline program arguments such as
node -e ${script}.

Fixes #37.

Solution

  • merge current main and use its context-aware rule that every interpolated
    value is one literal argument, equivalent to a quoted shell variable
  • remove the obsolete multiline/base64 virtual-writer changes during conflict
    resolution rather than maintain a second quoting path
  • preserve normal shell behavior, including platform-specific echo option
    handling; use portable printf '%s' when no trailing newline is required
  • drain Rust child stdout and stderr concurrently as byte chunks, avoiding pipe
    deadlocks and preserving whether either stream ended with a newline
  • snapshot JavaScript shell settings when an external invocation starts so
    concurrent test cleanup cannot change an in-flight command's error behavior
  • document printf '%s', stdin, and direct filesystem writes for exact text or
    binary output, with an executable JavaScript example
  • retain the existing quoteContext, preQuotedPassthrough, and raw controls;
    no multiline-specific compatibility mode is added

The full issue payload now survives unchanged as data. Shell substitutions in
interpolated values do not execute.

The source-level capture change is intentionally Rust-only because the
JavaScript runner already preserves missing final newlines. The PR therefore
uses the repository's parity-exempt label; behavioral parity is asserted in
both language test suites.

Reproduction and regression coverage

New JavaScript and Rust suites cover:

  • the complete issue payload with backticks, quotes, dollar signs, backslashes,
    and embedded newlines
  • echo and echo -n shell parity, plus portable printf '%s' exact output
  • redirection, append, paths containing spaces, and stdin writes
  • the reported quoted echo redirection form
  • multiline node -e / nested-shell programs remaining one argument
  • command-substitution and command-separator injection attempts
  • differential parity with /bin/sh using a quoted variable
  • exact Rust stdout and stderr capture without invented final newlines, using
    portable Unix and Windows fixtures
  • timeout-bounded concurrent capture with stderr output exceeding typical pipe
    capacity, guarding against stdout/stderr drain deadlocks

experiments/issue-37-multiline-competitors.mjs provides a reproducible
comparison with /bin/sh, command-stream, Bun Shell, zx, and execa. All five
matched for the tested multiline payload.

This is not a UI change, so screenshots do not apply.

Verification

  • bun run --cwd js lint
  • bun run --cwd js format:check
  • bun run --cwd js check:duplication
  • bun test js/tests/ --timeout 10000 — 1,421 passed, 6 skipped, 0 failed
  • Node 20 ESM/CommonJS load checks and both Node artifact test files — passed
  • cargo fmt --all -- --check
  • RUSTFLAGS=-Dwarnings cargo clippy --all-targets --all-features -- -D warnings
  • RUSTDOCFLAGS=-Dwarnings cargo doc --no-deps --all-features
  • cargo test --all-features plus doc tests — passed
  • release build, cargo package --allow-dirty, inline release-script tests,
    file-size check, and crate-size check — passed
  • JavaScript changeset and Rust changelog-fragment checks — passed
  • multiline example and competitor experiment — passed

Two fixed-wait SIGINT tests also surfaced as flaky under the complete suite;
both now wait for the child process's readiness message before signaling, and
failed fixtures clean up their child processes.

Adding CLAUDE.md with task information for AI processing.
This file will be removed when the task is complete.

Issue: #37
@konard konard self-assigned this Sep 9, 2025
konard and others added 2 commits September 9, 2025 21:06
Resolves #37: Multi-line strings containing backticks, dollar signs, and quotes
are now properly handled without corruption when used with echo commands.

**Changes:**
- Enhanced quote() function to detect complex multi-line strings
- Updated regex patterns to handle multi-line content (added 's' flag)
- Added virtual command bypass for complex echo > file patterns
- Created _write_multiline_content virtual command for safe file writing
- Added comprehensive test suite for various special character scenarios

**Technical Details:**
- Complex multi-line strings (with newlines + backticks/dollar signs) are
  automatically detected and processed via virtual command to bypass shell
  interpretation entirely
- Simple strings continue to work as before (backward compatible)
- Content is base64-encoded when passed to virtual command to prevent corruption

**Test Coverage:**
- Multi-line strings with mixed special characters ✅
- README files, configuration files, code examples ✅
- Backticks, dollar signs, quotes preservation ✅
- Backward compatibility with simple strings ✅

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@konard konard changed the title [WIP] Multi-line strings with special characters cause shell escaping problems Fix multi-line string escaping issues with special characters Sep 9, 2025
🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@konard
konard marked this pull request as ready for review September 9, 2025 18:23
@konard

konard commented Sep 13, 2026

Copy link
Copy Markdown
Member Author

We should double check more cases similar like this, make sure we have test coverage similar to all our competitors, and select behavior closer to how it would behave in sh scripts or with least surprise based on best practices from competitors. If there multiple options we should allow to configure, and use closer to sh behavior by default.

@konard
konard marked this pull request as draft September 13, 2026 15:04
@konard

konard commented Sep 13, 2026

Copy link
Copy Markdown
Member Author

🤖 AI Work Session Started

Starting automated work session at 2026-09-13T15:04:14.990Z

The PR has been converted to draft mode while work is in progress.

This comment marks the beginning of an AI work session. Please wait for the session to finish, and provide your feedback.

@konard

konard commented Sep 13, 2026

Copy link
Copy Markdown
Member Author

Working session summary

Investigation is complete; no code has been changed yet. The branch remains clean.

Root causes:

  • PR 111’s multiline special case writes files directly through _write_multiline_content, changing normal echo behavior.
  • For every other multiline command, it inserts the value unquoted. This causes the recorded CI failure at yes-command-cleanup.test.mjs:171: /bin/sh executes the embedded JavaScript as shell syntax.
  • The archived CI logs expired, but GitHub annotations preserved the failure. It reproduces locally: 6 pass, 1 fail.
  • Current main already fixes the underlying problem generically through context-aware literal interpolation from PR 100 and PR 201. The failing test passes there, as does the exact issue reproduction.

Recommended design:

1. Merge current `main` and resolve conflicts by retaining its JS/Rust architecture and generic quoting implementation.
2. Remove the obsolete base64 virtual-writer approach from this PR.
3. Preserve shell-like defaults:
   - Interpolated multiline text remains one literal argument.
   - `$`, backticks, quotes, backslashes, and newlines are not re-evaluated.
   - `echo` adds its normal trailing newline.
   - Document `printf %s` or filesystem/stdin APIs when byte-exact writing is required.
4. Reuse existing compatibility controls (`quoteContext`, `preQuotedPassthrough`, and `raw`) rather than introduce another multiline-specific mode.
5. Add issue-focused JS and Rust regression coverage for:
   - The full reported special-character example.
   - `echo`, `echo -n`, `printf`, append redirection, and paths with spaces.
   - The previously failing `node -e ${script}` case.
   - Injection attempts and `/bin/sh` differential parity.
6. Add a reproducible competitor experiment covering the repository’s named competitors: Bun, zx, and execa. Their documented defaults likewise treat interpolated values as literal data; zx and Bun provide explicit escape hatches. [Bun Shell](https://bun.sh/docs/runtime/shell), [zx quoting](https://google.github.io/zx/quotes), [execa](https://github.com/sindresorhus/execa).
7. Update JS/Rust best-practice documentation and add the required release fragments.
8. Run all local JS/Rust CI-equivalent checks, commit atomically, push only `issue-37-e875842b`, update [PR 111](https://github.com/link-foundation/command-stream/pull/111), verify fresh CI timestamps/SHA, review the final diff, and mark it ready.

Alternative designs—porting the virtual writer or introducing a second multiline mode—would duplicate current quoting machinery and produce less shell-like behavior.

Please reply yes to approve this design and allow implementation.


This summary was automatically extracted from the AI working session output.

@konard
konard marked this pull request as ready for review September 13, 2026 15:13
@konard

konard commented Sep 13, 2026

Copy link
Copy Markdown
Member Author

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

  • Model: GPT-5.6 Sol
  • Provider: OpenAI
  • Public pricing estimate: $3.403969

📊 Context and tokens usage:

  • 151.7K / 200K (76%) input tokens, 15.9K / 128K (12%) output tokens

Total: (151.7K + 2.1M cached) input tokens, 15.9K output tokens, $3.403969 cost

🤖 Models used:

  • Tool: OpenAI Codex
  • Requested: gpt-5.6-sol
  • Thinking level: xhigh (~31999 tokens)
  • Model: GPT-5.6 Sol (gpt-5.6-sol)

📎 Log file uploaded as Gist (1631KB)


Now working session is ended, feel free to review and add any feedback on the solution draft.

@konard

konard commented Sep 13, 2026

Copy link
Copy Markdown
Member Author

🔄 Auto-restart 1/5

Reason: CI failures detected; Merge conflicts

Starting new session to address the issues.


Auto-restart-until-mergeable mode is active. This run will stop after 5 restart iterations in total.

@konard
konard marked this pull request as draft September 13, 2026 15:15
konard and others added 6 commits September 13, 2026 15:25
Drain stdout and stderr concurrently as bytes so capture keeps missing final newlines intact.\n\nRefs #37
Cover stdin and competitor behavior alongside executable exact-write examples.\n\nRefs #37
Document printf and stdin guidance and add patch release metadata.\n\nRefs #37
Resolve documentation conflicts while preserving JSON and multiline guidance.
Use the child's setup output instead of a fixed delay under full-suite load.
@konard konard changed the title Fix multi-line string escaping issues with special characters Fix multiline interpolation and exact Rust output Sep 13, 2026
@konard konard added the parity-exempt Change is intentionally single-language; skip JS/Rust parity check label Sep 13, 2026
@konard
konard marked this pull request as ready for review September 13, 2026 16:28
@konard

konard commented Sep 13, 2026

Copy link
Copy Markdown
Member Author

Working session summary

Resolved and pushed to PR #111.

  • Merge conflicts resolved against latest main.
  • Fixed exact Rust stdout/stderr capture and pipe deadlock handling.
  • Fixed JavaScript in-flight shell-setting race.
  • Added comprehensive multiline, portability, and regression tests.
  • All 27 executable CI checks pass on commit 5fb8388; expected release/parity jobs were skipped.
  • PR is mergeable, clean, and ready for review.
  • Working tree is clean and matches the remote branch.

This summary was automatically extracted from the AI working session output.

@konard

konard commented Sep 13, 2026

Copy link
Copy Markdown
Member Author

🔄 Auto-restart-until-mergeable Log 1/5

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

  • Model: GPT-5.6 Sol
  • Provider: OpenAI
  • Public pricing estimate: $40.955098

📊 Context and tokens usage:

  • 963.4K / 200K (482%) input tokens, 128.6K / 128K (100%) output tokens

Total: (963.4K + 36.7M cached) input tokens, 128.6K output tokens, $40.955098 cost

🤖 Models used:

  • Tool: OpenAI Codex
  • Requested: gpt-5.6-sol
  • Model: GPT-5.6 Sol (gpt-5.6-sol)

📎 Log file uploaded as Gist (16960KB)


Now working session is ended, feel free to review and add any feedback on the solution draft.

@konard
konard merged commit 5d022d2 into main Sep 13, 2026
34 checks passed
@konard

konard commented Sep 13, 2026

Copy link
Copy Markdown
Member Author

🎉 Auto-merged

This pull request has been automatically merged by hive-mind.

  • All CI checks have passed

Auto-merged by hive-mind with --auto-merge flag

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parity-exempt Change is intentionally single-language; skip JS/Rust parity check

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multi-line strings with special characters cause shell escaping problems

1 participant