Repository navigation
feat(vended-tools): add python_repl vended tool to TS [replica of harness-sdk PR 5041] #35
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
poshinchen
wants to merge
3
commits into
base-pr-5041
Choose a base branch
from
strandly-v3-pr-5041
base: base-pr-5041
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| # Python REPL Tool | ||
|
|
||
| Runs model-generated Python through a [Sandbox](../../sandbox/base.ts) by calling `sandbox.executeCode(code, 'python3')`. | ||
|
|
||
| Each call runs in a fresh interpreter, so variables, imports, and definitions do not persist across calls. Files written to the sandbox filesystem persist for as long as the sandbox does, so the model can checkpoint intermediate results to disk and load them in a later call. stdin is not available, so `input()` fails. | ||
|
|
||
| > **Security warning**: this tool executes arbitrary Python code. Without an isolating sandbox (the default `NotASandboxLocalEnvironment`), code runs with the full permissions of the process. | ||
|
|
||
| ## Usage | ||
|
|
||
| ```typescript | ||
| import { Agent } from '@strands-agents/sdk' | ||
| import { pythonRepl } from '@strands-agents/sdk/vended-tools/python-repl' | ||
|
|
||
| const agent = new Agent({ tools: [pythonRepl] }) | ||
| await agent.invoke('Use python_repl to compute the first 10 Fibonacci numbers and print them.') | ||
| ``` | ||
|
|
||
| With a bound sandbox or a custom interpreter: | ||
|
|
||
| ```typescript | ||
| import { DockerSandbox } from '@strands-agents/sdk/sandbox/docker' | ||
| import { makePythonRepl } from '@strands-agents/sdk/vended-tools/python-repl' | ||
|
|
||
| const pythonTool = makePythonRepl(new DockerSandbox({ container: 'my-container' }), { language: 'python3.12' }) | ||
| ``` | ||
|
|
||
| Without a bound sandbox, the tool reads `context.agent.sandbox` at call time. | ||
|
|
||
| ## API | ||
|
|
||
| ### `makePythonRepl(options?)` / `makePythonRepl(sandbox, options?)` | ||
|
|
||
| | Option | Type | Default | Description | | ||
| | ------------- | -------- | ------------- | ------------------------------------------------------------------------------------------ | | ||
| | `name` | `string` | `python_repl` | Tool name. Must be non-empty. | | ||
| | `description` | `string` | (built-in) | Description shown to the model. | | ||
| | `language` | `string` | `python3` | Interpreter on the sandbox's `PATH`. Validated against `LANGUAGE_PATTERN` at construction. | | ||
|
|
||
| ### Input | ||
|
|
||
| ```typescript | ||
| { | ||
| code: string // Python source to execute | ||
| timeout?: number // Timeout in seconds (default: 120). Must be positive. | ||
| } | ||
| ``` | ||
|
|
||
| ### Return Value | ||
|
|
||
| ```typescript | ||
| interface PythonReplOutput { | ||
| output: string // Standard output (stdout) | ||
| error: string // Standard error (stderr), including tracebacks - empty string if none | ||
| exit_code: number // Exit code of the interpreter - non-zero means the code failed | ||
| } | ||
| ``` | ||
|
|
||
| ### Error Handling | ||
|
|
||
| - Errors in the Python code itself are reported through `error` and a non-zero `exit_code`, not thrown. | ||
| - `SandboxTimeoutError`: execution exceeded `timeout`. The message carries the partial output as JSON with the success field names and `exit_code: 124`. | ||
| - `SandboxAbortError`: the agent cancelled the call (`context.cancelSignal`); the sandbox is told to stop the interpreter and the error propagates unwrapped. | ||
| - `PythonReplError`: the sandbox failed to run the code (for example, an unreachable container). The original error is available as `cause`. |
162 changes: 162 additions & 0 deletions
162
strands-ts/src/vended-tools/python-repl/__tests__/python-repl.test.node.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,162 @@ | ||
| import { describe, it, expect, vi } from 'vitest' | ||
| import { mkdtempSync } from 'fs' | ||
| import { tmpdir } from 'os' | ||
| import { join } from 'path' | ||
| import { | ||
| makePythonRepl, | ||
| pythonRepl, | ||
| PYTHON_REPL_DESCRIPTION, | ||
| PythonReplError, | ||
| type PythonReplOutput, | ||
| } from '../index.js' | ||
| import * as vendedTools from '../../index.js' | ||
| import { NotASandboxLocalEnvironment } from '../../../sandbox/not-a-sandbox-local-environment.js' | ||
| import { SandboxAbortError, SandboxTimeoutError } from '../../../sandbox/errors.js' | ||
| import type { Sandbox } from '../../../sandbox/base.js' | ||
| import type { ToolContext } from '../../../index.js' | ||
| import { createMockAgent } from '../../../__fixtures__/agent-helpers.js' | ||
|
|
||
| const toolContext = ( | ||
| sandbox: Sandbox = new NotASandboxLocalEnvironment(), | ||
| cancelSignal = new AbortController().signal | ||
| ): ToolContext => { | ||
| const agent = createMockAgent({ extra: { sandbox } }) | ||
| return { | ||
| toolUse: { name: 'python_repl', toolUseId: 'id', input: {} }, | ||
| agent, | ||
| invocationState: {}, | ||
| cancelSignal, | ||
| interrupt: () => { | ||
| throw new Error('interrupt not available in mock context') | ||
| }, | ||
| } | ||
| } | ||
|
|
||
| const mockSandbox = ( | ||
| impl: () => Promise<unknown> = async () => ({ type: 'executionResult', exitCode: 0, stdout: 'hi\n', stderr: '' }) | ||
| ) => { | ||
| const sandbox = new NotASandboxLocalEnvironment() | ||
| const executeCode = vi.spyOn(sandbox, 'executeCode').mockImplementation(impl as Sandbox['executeCode']) | ||
| return { sandbox, executeCode } | ||
| } | ||
|
|
||
| const run = (input: { code: string; timeout?: number }, context: ToolContext, tool = pythonRepl) => | ||
| tool.invoke(input, context) as Promise<PythonReplOutput> | ||
|
|
||
| describe('python_repl shim', () => { | ||
| it('uses the agent sandbox and defaults', async () => { | ||
| const { sandbox, executeCode } = mockSandbox() | ||
| const context = toolContext(sandbox) | ||
| const result = await run({ code: "print('hi')" }, context) | ||
| expect(executeCode).toHaveBeenCalledExactlyOnceWith("print('hi')", 'python3', { | ||
| timeout: 120, | ||
| signal: context.cancelSignal, | ||
| }) | ||
| expect(result).toStrictEqual({ output: 'hi\n', error: '', exit_code: 0 }) | ||
| }) | ||
|
|
||
| it('prefers a bound sandbox and forwards language and timeout', async () => { | ||
| const bound = mockSandbox() | ||
| const agentSandbox = mockSandbox() | ||
| const tool = makePythonRepl(bound.sandbox, { language: 'python3.12' }) | ||
| await run({ code: 'pass', timeout: 7 }, toolContext(agentSandbox.sandbox), tool) | ||
| expect(bound.executeCode).toHaveBeenCalledExactlyOnceWith( | ||
| 'pass', | ||
| 'python3.12', | ||
| expect.objectContaining({ timeout: 7 }) | ||
| ) | ||
| expect(agentSandbox.executeCode).not.toHaveBeenCalled() | ||
| }) | ||
|
|
||
| it('rejects a non-positive timeout', async () => { | ||
| const { sandbox, executeCode } = mockSandbox() | ||
| await expect(run({ code: 'pass', timeout: 0 }, toolContext(sandbox))).rejects.toThrow( | ||
| /timeout must be a positive number of seconds, got 0/ | ||
| ) | ||
| expect(executeCode).not.toHaveBeenCalled() | ||
| }) | ||
|
|
||
| it.each([ | ||
| [new Error('container gone'), 'container gone'], | ||
| ['container gone', 'container gone'], | ||
| ])('wraps sandbox error %o as PythonReplError', async (boom, message) => { | ||
| const { sandbox } = mockSandbox(async () => { | ||
| throw boom | ||
| }) | ||
| const error = await run({ code: 'pass' }, toolContext(sandbox)).catch((e: unknown) => e) | ||
| expect(error).toBeInstanceOf(PythonReplError) | ||
| expect(error).toMatchObject({ message, cause: boom }) | ||
| }) | ||
|
|
||
| it('requires a tool context', async () => { | ||
| await expect(pythonRepl.invoke({ code: 'pass' })).rejects.toThrow('Tool context is required') | ||
| }) | ||
| }) | ||
|
|
||
| describe.skipIf(process.platform === 'win32')('python_repl local execution', () => { | ||
| it.each([ | ||
| ["raise ValueError('bad')", 'ValueError: bad'], | ||
| ['input()', 'EOFError'], | ||
| ])('reports failure of %s via error and non-zero exit', async (code, expectedError) => { | ||
| const result = await run({ code }, toolContext()) | ||
| expect(result.exit_code).not.toBe(0) | ||
| expect(result.error).toContain(expectedError) | ||
| }) | ||
|
|
||
| it('does not persist state across calls but files do', async () => { | ||
| const path = join(mkdtempSync(join(tmpdir(), 'python-repl-test-')), 'state.json') | ||
| const ctx = toolContext() | ||
| await run({ code: `import json; x = 42; json.dump({'x': x}, open(${JSON.stringify(path)}, 'w'))` }, ctx) | ||
|
|
||
| const missing = await run({ code: 'print(x)' }, ctx) | ||
| expect(missing.error).toContain('NameError') | ||
|
|
||
| const restored = await run({ code: `import json; print(json.load(open(${JSON.stringify(path)}))['x'])` }, ctx) | ||
| expect(restored).toStrictEqual({ output: '42\n', error: '', exit_code: 0 }) | ||
| }) | ||
|
|
||
| it('timeout carries partial output with the success field names', async () => { | ||
| const error = await run( | ||
| { code: "print('partial', flush=True)\nimport time; time.sleep(10)", timeout: 0.5 }, | ||
| toolContext() | ||
| ).catch((e: unknown) => e) | ||
| expect(error).toBeInstanceOf(SandboxTimeoutError) | ||
| const payload = JSON.parse((error as Error).message.split('\n')[1]!) | ||
| expect(payload).toStrictEqual({ output: 'partial\n', error: '', exit_code: 124 }) | ||
| }) | ||
|
|
||
| it('cancellation stops the call and propagates unwrapped', async () => { | ||
| const controller = new AbortController() | ||
| const pending = run({ code: 'import time; time.sleep(10)' }, toolContext(undefined, controller.signal)) | ||
| setTimeout(() => controller.abort(), 100) | ||
| const start = Date.now() | ||
| await expect(pending).rejects.toBeInstanceOf(SandboxAbortError) | ||
| expect(Date.now() - start).toBeLessThan(5000) | ||
| }) | ||
| }) | ||
|
|
||
| describe('makePythonRepl', () => { | ||
| it.each([ | ||
| [{ name: '' }, /name/], | ||
| [{ language: 'python3; rm -rf /' }, /language/], | ||
| ])('rejects invalid arguments %o', (options, match) => { | ||
| expect(() => makePythonRepl(options)).toThrow(match) | ||
| }) | ||
|
|
||
| it('has the default tool spec and exports', () => { | ||
| expect(pythonRepl.name).toBe('python_repl') | ||
| expect(pythonRepl.toolSpec.description).toBe(PYTHON_REPL_DESCRIPTION) | ||
| const schema = pythonRepl.toolSpec.inputSchema as { properties: Record<string, unknown>; required: string[] } | ||
| expect(Object.keys(schema.properties).sort()).toStrictEqual(['code', 'timeout']) | ||
| expect(schema.required).toStrictEqual(['code']) | ||
| expect(vendedTools.pythonRepl).toBe(pythonRepl) | ||
| expect(vendedTools.makePythonRepl).toBe(makePythonRepl) | ||
| }) | ||
|
|
||
| it('resolves both overloads and applies a custom name and description', () => { | ||
| const tool = makePythonRepl(undefined, { name: 'run_python', description: 'custom' }) | ||
| expect(tool.name).toBe('run_python') | ||
| expect(tool.toolSpec.description).toBe('custom') | ||
| expect(makePythonRepl(new NotASandboxLocalEnvironment()).name).toBe('python_repl') | ||
| }) | ||
| }) | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Suggestion
Issue: The timeout test asserts the partial payload by splitting the error message on
\nand parsing index[1]. This couples the test to the exact message layout, so any formatting change (even an extra newline in the base message) breaks it in a non-obvious way.Suggestion: If the partial output were exposed as a structured field on the error (see comment in python-repl.ts), this assertion could read that field directly instead of parsing the message string.