writing-great-skills defines a no-op as "a line the model already obeys by default, so you pay load to say nothing. The test: does it change behavior versus the default?"

I've been running that exact test at whole-skill granularity with a harness instead of judgment. When you look at the results you'll understand why I felt compelled to share.

The setup: take a task from exactly the situation a skill was written for, give it to a current frontier model (claude-sonnet-5) without the skill — at least three independent runs per task — scored by a deterministic check, no LLM judge. If the model passes unaided every time, the skill is a no-op at skill scale: there's nothing left for it to improve. The protocol was pre-registered before any run: https://github.com/MrBinnacle/skill-harness/blob/main/docs/findings/v0.2-preregistration.md

The result so far: 26 out of 26 unaided runs passed, across 6 independently-authored tasks — including four skills of my own that I'd previously assumed were adding value. All four were screened before admission to my collection, and none passed muster. The full records — pre-registration, per-screen results, and the retirement log — are public: https://github.com/MrBinnacle/skills/blob/main/RETIRED.md

My 2 main takeaways: unsettling feeling that those skills felt necessary but would have silently continued to cost more and add less value over time - the current models just have the reasoning capability now. Second, the measurement gets structurally hard exactly where skills feel most valuable: on well-specified self-contained tasks, frontier models are at ceiling, so with-vs-without differences can be unmeasurable in principle, not just in practice. Written up here: https://github.com/MrBinnacle/skill-harness/blob/main/docs/case-studies/double-ceiling-structurally-unmeasured.md

Sharing because the no-op test is the sharpest framing I've seen of what this harness measures, and the data says it bites much harder than intuition expects. If any of it is useful for writing-great-skills, take whatever you want.

Thank you Matt - your work continues to be my most trusted reference for what 'right' looks like in this domain.