Skip to content

Agent mcp server - #1519

Draft
Gin890 wants to merge 20 commits into
mainfrom
agent-mcp-server
Draft

Agent mcp server#1519
Gin890 wants to merge 20 commits into
mainfrom
agent-mcp-server

Conversation

@Gin890

@Gin890 Gin890 commented Sep 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Used Claude to create a Python interface to connect to the mincrocontroller and control Switch. It also created an MCP server for AI agent use as part of the Python codebase.

It also adds a new automation program in the C++ app to act as a MCP server so users can monitor and intervene agent control of Switch.

From now on, if you want to automate sth unimplemented, just pay Anthropic/OpenAI to let agent play it for you 😜

@Mysticial
Mysticial marked this pull request as draft September 26, 2026 23:07
@Mysticial

Mysticial commented Sep 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

No chance this is going in as-is. Way too big. Way too much slop. We'll need to break this into many smaller PRs to incrementally build it and understand every step of it. Like why is there Python here?

But we can keep this PR for reference or if anyone wants to try it.

Gin and others added 20 commits October 6, 2026 21:27
Gin @claude
PybindSwitchProController: log to the global logger instead of stdout
ad19f2a 
The controller logged through global_logger_command_line(), which also prints
every line to stdout. Python callers, and MCP servers that use stdio for their
protocol, need stdout to stay clean. Log to global_logger_raw() instead; lines
still reach every listener of the global logger, e.g. SerialProgramsCommandLine's
log file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin @claude
PybindSwitchProController: throw when a command is sent before the co…
b32feb3 
…ntroller is ready

Commands used to log "Controller is not ready." and silently do nothing, so a
caller couldn't tell that its input was dropped. Throw
InvalidConnectionStateException with the connection status instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin @claude
PybindSwitchProController: return the connection status as plain text
0d57a3a 
current_status() returned ControllerConnection::status_text(), which is
formatted for the GUI: each line wrapped in <font color=...> tags, lines
joined with <br>.

Add ControllerConnection::raw_status_text(), which keeps the text of each
status line as it was set (without colors) and joins the lines with a newline,
and use it in current_status().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin @claude
PybindSwitchProController: add controller_name() and cancel_all()
54ef918 
- controller_name(): the active controller implementation, e.g.
  "Nintendo Switch: Pro Controller".
- cancel_all(): drop queued commands and release all inputs. Safe to call
  from another thread while one is blocked in wait_for_all_requests().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin @claude
Controllers: add release_all_and_confirm()
d051aab 
cancel_all_commands() only starts the cancellation: it returns as soon as the
cancel request is handed to the connection, and the host marks its command
queue empty at that moment, so waiting for the queue proves nothing.

release_all_and_confirm() cancels, queues a 10 ms neutral no-op, and waits
(bounded by a timeout) for the device to report that the no-op finished.
Commands are delivered in order, so that report means the device has processed
the cancel and is holding the neutral state. Uses only the AbstractController
interface, so it works for any controller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin @claude
PybindSwitchProController: add release_all()
09767df 
cancel_all() plus a bounded wait for the device to confirm the neutral state,
using release_all_and_confirm(). Returns false on timeout, e.g. when the Switch
is asleep and the device can't execute commands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin @claude
Python: add the pokemon_automation package and its input vocabulary
5cb622d 
Start the pokemon_automation Python package (SerialPrograms/Source/PythonBindings/).

- _core.py: finds and imports the compiled _pa_core module (added next) lazily,
  so the pure-Python parts work without it.
- buttons.py: the input vocabulary scripts and AI agents use: button names and
  aliases ("A", "L+R", "start", "L3"), d-pad directions ("up", "up-right"), and
  joystick positions (direction names or [x, y], +y = up). Bit values match
  NintendoSwitch::Button and DpadPosition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin @claude
Add the _pa_core pybind11 module (CMake option PA_PYTHON_BINDINGS)
90c4526 
_pa_core wraps PybindSwitchProController for Python. It links only CoreLib (no
Qt, OpenCV or Tesseract), so it also builds with -DPA_CORE_ONLY=ON.

- CMake option PA_PYTHON_BINDINGS (OFF by default, since it needs a Python
  interpreter with development headers), in both the full and the core-only
  build. pybind11 is taken from the environment or downloaded.
- The built module is copied into the pokemon_automation package folder.
- A log sink that never writes to stdout (stdout carries the protocol for MCP
  over stdio), with recent lines available to Python.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin @claude
Python: add SwitchController and input steps
cdf1dba 
- SwitchController: press buttons/d-pad, tilt sticks, hold combinations, run
  sequences of InputSteps, flush, stop and release_all.
- InputStep: one step of a sequence (buttons, sticks, hold, release, repeat,
  wait). The same step dicts are used by the MCP run_inputs tool.
- FakeController: records commands, for tests without hardware.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin @claude
Python: add video capture and OCR
d827f8f 
Pure Python, outside the C++ codebase:
- video.py: continuous capture with opencv-python (only the latest frame is
  kept, so a frame taken after an input shows its effect), JPEG/PNG encoding,
  cropping with ImageFloatBox-style boxes, OCR with pytesseract (optional).
- devices.py: list serial ports and video devices (device names on macOS via
  pyobjc AVFoundation, in OpenCV's index order).
- FakeVideoCapture for tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin @claude
Python: add Console and the hardware self-test
11aab04 
- Console: a controller plus a video source. act() sends inputs, waits until the
  device has executed them, waits for the game to react, then returns a frame
  captured after all of that.
- selftest.py: `python -m pokemon_automation.selftest --serial ... --video ...`
  checks real hardware with harmless inputs and saves screenshots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin @claude
Add the AI agent tool definitions (AgentTools.json)
ebd8a96 
The MCP interface for AI agents that control the Switch, as data, so the Python
MCP server and the upcoming SerialPrograms app server expose exactly the same
tools: names, descriptions, JSON Schemas of the arguments, which host implements
each tool, and the instructions sent to agents.

AgentInputTestCases.json lists inputs (buttons, sticks, steps) and their
expected parse results or errors; both the Python and C++ parsers are tested
against it.

Also: agent_tools.py loads the file (inlining $refs), tests that the Python
input vocabulary passes the shared cases, and the build copies both files into
the Python package.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin @claude
Python: add the MCP server
0f9593c 
pokemon_automation.mcp_server serves a Console to AI agents over MCP (stdio or
streamable HTTP). Tools, descriptions and schemas come from AgentTools.json;
tests check that the Python functions accept exactly the shared arguments.

Input tools return a screenshot taken after the game reacts; release_all works
while another call is running; per-call input limits; read-only mode; --fake
runs without hardware.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin @claude
Python: add a README
7a464f3 
How to build _pa_core, install the package, run the self-test, use the Python
API and the MCP server, and run the tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin @claude
AgentServer: add a minimal HTTP server on QTcpServer
d02e4bd 
The app needs a local HTTP endpoint for AI agents (MCP's Streamable HTTP
transport). Qt's QHttpServer module is GPL-3.0-only, which this project doesn't
take, so this is a small HTTP/1.1 server on QTcpServer (Qt Network, LGPL):
Content-Length bodies, keep-alive and "Expect: 100-continue"; no chunked
requests or pipelining.

The listener and sockets live on their own thread with its own event loop;
each request is handled on a worker thread, so a slow request doesn't block
others. stop() closes connections and waits for running handlers.

Qt Network is now listed explicitly. The app already used it (FileDownloader,
DiscordWebhook) but only got it through other Qt modules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin @claude
AgentServer: load the tool definitions from AgentTools.json
02aa4e6 
- AgentTools.json is compiled into SerialProgramsLib with a new CMake helper,
  pa_embed_text_file() (cmake/EmbedTextFile.cmake), as a byte array, so it isn't
  limited by compilers' string-literal size limits.
- AgentToolDefinitions parses it, keeps the tools for host "app", inlines
  $refs (same as the Python loader), and validates tool arguments against the
  schemas with a small validator for the keywords the file uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin @claude
AgentServer: parse input steps
d39f930 
Parse the input vocabulary of AgentTools.json (button names, d-pad, sticks,
input steps) into NintendoSwitch::Button, DpadPosition and JoystickPosition.
The C++ twin of pokemon_automation/buttons.py; both pass the shared
AgentInputTestCases.json.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin @claude
AgentServer: add the MCP protocol server
d467eb4 
MCP over Streamable HTTP at POST /mcp: JSON-RPC 2.0 with the initialize
handshake (protocol versions 2024-11-05 to 2025-11-25; newer clients' probe
gets "method not found" and falls back to it), ping, tools/list and tools/call,
sessions, and plain JSON responses.

Security: an optional bearer token, and DNS-rebinding protection (Origin and
Host checks). The tools themselves are implemented by an McpToolHandler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin @claude
GameConsole: tell listeners about the user's controller input
034b96e 
- ConsoleSystemSession::Listener gets on_controller_input(), called after the
  user's keyboard input was sent to the console's controllers (not for input
  suppressed because the console isn't focused or its controllers are locked).
  It runs outside the session lock; the forwarding code itself is unchanged.
- ConsoleHandle::system_session() gives programs access to their session, so
  they can register such a listener.

Used by the AI Agent Server program to notice when the user takes over.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gin @claude
ML: add the AI Agent Server program
b940dba 
A program in the ML tab (developer mode only) that hosts the MCP server, so AI
agents (Claude Code, Codex, ...) can control the Switch through SerialPrograms
while the user watches. Agent inputs run on the program's controller context;
screenshots come from its video feed; read_text uses the app's Tesseract OCR.

Pressing a key hands control to the user: running agent inputs are interrupted
and new ones refused until the user clicks "Return control to agent" (or after
an optional idle time). Stop shuts the server down and releases the controller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

PR: AI-code-we-don't-understand

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Gin890 @Mysticial