[OMEGA-421] Install each plugin's Python dependencies - #372
Open
surafelfikru wants to merge 6 commits into
Open
surafelfikru wants to merge 6 commits into
surafelfikru wants to merge 6 commits into
Conversation
Core has a plugins/ directory, a manifest, a loader and a documented plugin API, and the one thing a plugin cannot state is what it needs to import. Only core's own requirements.txt reaches pip, so a plugin's imports are satisfied by accident or not at all: plugins/openclaw/openclaw.py imports requests, which appears in no requirements file and arrives transitively through chromadb's kubernetes client. Neither in-tree plugin ships a requirements.txt, because nothing would read one. scripts/install_dependencies.sh walks plugins/*/requirements.txt and installs whatever it finds alongside core's own. Core never names a plugin, and a plugin that declares nothing costs nothing. Everything resolves in one pip invocation. Separate invocations resolve separately, so a plugin pinning a version core also pins would replace core's copy in silence — core is not a pip distribution, so no metadata records what it needed and nothing warns. Together, a real conflict fails the install, and pip check afterwards catches what resolution let through. The walk lives in a script rather than the Dockerfile because the image is not the only way Omega is installed. The README's source install had the same gap: it installed core's requirements and no plugin's. Both paths call the script, which takes pass-through pip options so the build can add --no-cache-dir and --break-system-packages while a venv install adds none. Torch keeps its own step because it comes from the CPU wheel index rather than PyPI, but its version is no longer written beside it. Both the Dockerfile and the README read the pin out of requirements.txt, which pins torch for the combined step as well. A second copy drifts, and the drift is silent: the CPU index serves 2.14.0+cpu today against a pin of 2.12.1, so the unpinned README command installed the CPU build and the next step replaced it with PyPI's CUDA one.
The plugin API reference described a plugin's three manifest fields and never mentioned dependencies, so a plugin author had nowhere to learn that a requirements.txt is read, or what happens when one fights Omega's own pins. That silence is how an in-tree plugin ended up importing a package nothing declares. The new section says where the file goes, that everything installs in one pip invocation so Omega's pins win, and to declare ranges wide enough to include them. It also names the two limits worth knowing: pinning a package Omega depends on indirectly changes it without any error, and a plugin mounted into an already built image gets no installation at all.
Both tests passed against the regression they exist to catch, so the protection they were adding was zero. The check that core names no plugin looked for the plugin's name among the whitespace-separated words of the Dockerfile. A hardcoded path does not appear that way: --mount=type=bind,source=plugins/openclaw,target=... is one word, and it is not "openclaw". Anyone reintroducing the very coupling the test exists to prevent would have gone through it. It now looks for the name on a word boundary, which finds it inside a longer argument. The check that the README installs the torch version core pins read only the line naming the CPU wheel index, and accepted that line unconditionally when it ended in a backslash — which it does, because the pin sits on the continuation line the test never looked at. Rewriting the pin to a bare "torch" left the first line untouched and the test green, and that is exactly the silent swap of the CPU build for PyPI's CUDA one it was written to stop. It now joins continuation lines first and reads the whole command. Both were confirmed by making each change and watching the suite stay green, then confirmed fixed by making it again and watching it fail.
Installing core and every plugin in one pip invocation is what makes core's pins win, and it is also what lets a plugin overrule them in a way no error reports. pip reads an option written inside a requirements file as an instruction for the whole invocation rather than for the file carrying it, so a plugin whose first line is an --index-url decides where every package in the install is fetched from. Core's own pinned packages included: with a plugin naming another index, pip asks that host for core's pyyaml. The obvious repair does not work. Passing --index-url on the command line does not defend the install, because pip applies the file's options to its finder while parsing, after the command line has already been read — the file wins. Nothing can be scoped to the plugin that shipped it either, since the whole point of the single invocation is that there is only one resolution. So the file is refused: any line that begins with a dash stops the install before pip runs, naming the file, the line number and the line, so whoever wrote the plugin can see what to remove. This narrows what a plugin may declare. A plugin needing a package from a private index cannot say so here and needs an image built on top of core instead. That capability was never really available — exercising it would have silently moved core's own downloads to the same index. The plugin API reference now states the rule next to the promise it was quietly contradicting, which told plugin authors that Omega's pins win.
openclaw.py imports requests, and until now no requirements file in the tree said so. It worked because chromadb pulls requests in for its kubernetes client, which is a dependency of a dependency and free to disappear: the day it does, the plugin fails to load and takes start-up with it. Nothing detects this automatically — pip check cannot, because core is not a pip distribution and so declares nothing for it to check against. The protection is the declaration itself. This is also the first plugin in the tree to use the mechanism, so the walk over plugins/*/requirements.txt now has something real to find rather than only its fixtures.
The install step mounted all of plugins/, so editing any plugin file reinstalled every Python package. A collector stage now copies only each plugin's requirements.txt into a stage of its own, and the install step mounts that instead. Checked with real builds: editing plugin code keeps the step cached, and changing any requirements file reruns it. A plugin requirement that points at a file shipped inside the plugin, such as a vendored wheel, no longer reaches the install step.
alyona-snet
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before this change, a plugin could not use a new Python package without editing core's requirements.txt. Now a plugin can put its own requirements.txt in its folder, and the new script scripts/install_dependencies.sh installs core's packages and all plugin packages in one pip command. The Docker build and the source install in the README both use this script. Because everything is installed together, core's versions win: if a plugin asks for a different version of a package that core pins, the install stops with an error instead of silently replacing core's version, and pip check runs at the end to find other problems. A plugin's file may only list packages, and lines with pip options such as --index-url are refused, because pip would apply them to core's packages too. The Docker build gives the install step only the plugins' requirements files, so changing plugin code does not reinstall everything; I checked this with real builds. There are some limits: a plugin mounted into an already built image gets nothing installed, and a requirement that points to a file inside the plugin folder, such as a bundled wheel, does not work in the Docker build. This PR also adds requests to the openclaw plugin, which imports it but did not declare it.
Currently WIP there might be some security issues i might have missed.