test: stop the suite from writing skill files into the real $HOME#527
Open
LucasRollo wants to merge 1 commit into
Open
test: stop the suite from writing skill files into the real $HOME#527LucasRollo wants to merge 1 commit into
LucasRollo wants to merge 1 commit into
Conversation
Running `pytest` mutated the developer's actual home directory.
`test_doctor_runs` calls the real `agent-reach doctor`, which as a side
effect auto-installs SKILL.md into every agent directory it finds
(~/.claude/skills, ~/.agents/skills, ~/.openclaw/skills) and creates
~/.agent-reach/ via Config(). The test patched nothing, so just running
the suite silently installed/overwrote skill files for the developer's
live AI agents — a surprising and potentially destructive side effect,
and the kind of thing a malicious dependency could weaponise.
- add tests/conftest.py: an autouse fixture redirecting HOME/USERPROFILE
to a throwaway tmp dir, and re-pointing Config.CONFIG_DIR/CONFIG_FILE
(class attributes bound to Path.home() at import time, before any
fixture runs) at that dir. Contains every home-directed write for the
whole suite, not just the one test that exposed it.
- add tests/test_home_isolation.py: asserts the isolation is in effect
so the leak can't regress unnoticed — if the fixture stops redirecting
HOME, expanduser("~") reverts to the real home and these fail.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Running the test suite mutates the developer's real home directory.
tests/test_cli.py::test_doctor_runsinvokes the realagent-reach doctor:_cmd_doctorcalls_install_skill(force=False)("Auto-install skill if not already present"), which installsSKILL.mdinto every agent directory it finds on the machine —~/.claude/skills/agent-reach/,~/.agents/skills/agent-reach/,~/.openclaw/skills/agent-reach/— andConfig()creates~/.agent-reach/. The test patches onlysys.argv, so all of that lands in the developer's real$HOME.That means simply running
pytestsilently installs (or overwrites) skill files for the developer's live AI agents. A SKILL.md is standing instructions for an agent, so this isn't just stray files — it can change how the developer's Claude Code / OpenClaw behaves afterward. It's a surprising side effect, and the same pattern a malicious dependency could use to inject agent instructions via a test run.The sibling test
test_doctor_preserves_existing_skill_installalready carefully sandboxesexpanduserand theConfigpaths, so the need for isolation is understood —test_doctor_runsjust lacked it, and nothing enforced it suite-wide.Fix
Rather than patch a single test, isolate
$HOMEfor the whole suite so no test can leak into the real home:tests/conftest.py— autouse fixture redirectingHOME/USERPROFILEto a throwaway tmp dir. It also re-pointsConfig.CONFIG_DIR/Config.CONFIG_FILE, which are class attributes bound toPath.home()at import time (before any fixture runs), so an env-var redirect alone wouldn't contain them.tests/test_home_isolation.py— asserts the isolation is actually in effect (expanduser + Config resolve under the sandbox; doctor's writes stay under it). If the fixture is ever removed or stops redirecting$HOME, these fail instead of silently leaking again.Verification
Full suite passes, and after running it the real
~/.claude/skills/agent-reachand~/.agent-reachare no longer created (previously they were).Note (out of scope for this PR)
Arguably a diagnostic command (
doctor) shouldn't install skills into the system at all as a side effect — that's a surprising mutation from a read-only-sounding command. Happy to follow up separately if the maintainers want to reconsider that behavior; this PR just stops the test suite from being affected by it.