Strengthen using-superpowers for explicit skill requests

obra · obra · commit 3dac35e0b35c · 2025-12-26T22:41:22.000-06:00
Addresses failure mode where Claude skips skill invocation even when
user explicitly requests it by name (e.g., "subagent-driven-development,
please").

Skill changes:
- "Check for skills" → "Invoke relevant or requested skills"
- "BEFORE ANY RESPONSE" → "BEFORE any response or action"
- Added reassurance that wrong skill invocations are okay
- New red flag: "I know what that means"

Tests:
- New test suite for explicit skill requests
- Single-turn and multi-turn scenarios with --continue
- Tests with haiku model and user CLAUDE.md
diff --git a/RELEASE-NOTES.md b/RELEASE-NOTES.md
@@ -1,5 +1,23 @@
 # Superpowers Release Notes
 
+## Unreleased
+
+### Improvements
+
+**Strengthened using-superpowers skill for explicit skill requests**
+
+Addressed a failure mode where Claude would skip invoking a skill even when the user explicitly requested it by name (e.g., "subagent-driven-development, please"). Claude would think "I know what that means" and start working directly instead of loading the skill.
+
+Changes:
+- Updated "The Rule" to say "Invoke relevant or requested skills" instead of "Check for skills" - emphasizing active invocation over passive checking
+- Added "BEFORE any response or action" - the original wording only mentioned "response" but Claude would sometimes take action without responding first
+- Added reassurance that invoking a wrong skill is okay - reduces hesitation
+- Added new red flag: "I know what that means" → Knowing the concept ≠ using the skill
+
+**Added explicit skill request tests**
+
+New test suite in `tests/explicit-skill-requests/` that verifies Claude correctly invokes skills when users request them by name. Includes single-turn and multi-turn test scenarios.
+
 ## v4.0.2 (2025-12-23)
 
 ### Fixes
diff --git a/skills/using-superpowers/SKILL.md b/skills/using-superpowers/SKILL.md
@@ -21,7 +21,7 @@ This is not negotiable. This is not optional. You cannot rationalize your way ou
 
 ## The Rule
 
-**Check for skills BEFORE ANY RESPONSE.** This includes clarifying questions. Even 1% chance means invoke the Skill tool first.
+**Invoke relevant or requested skills BEFORE any response or action.** Even a 1% chance a skill might apply means that you should invoke the skill to check. If an invoked skill turns out to be wrong for the situation, you don't need to use it.
 
 ```dot
 digraph skill_flow {
@@ -62,6 +62,7 @@ These thoughts mean STOP—you're rationalizing:
 | "The skill is overkill" | Simple things become complex. Use it. |
 | "I'll just do this one thing first" | Check BEFORE doing anything. |
 | "This feels productive" | Undisciplined action wastes time. Skills prevent this. |
+| "I know what that means" | Knowing the concept ≠ using the skill. Invoke it. |
 
 ## Skill Priority
 
diff --git a/tests/explicit-skill-requests/prompts/action-oriented.txt b/tests/explicit-skill-requests/prompts/action-oriented.txt
@@ -0,0 +1,3 @@
+The plan is done. docs/plans/auth-system.md has everything.
+
+Do subagent-driven development on this - start with Task 1, dispatch a subagent, then we'll review.
diff --git a/tests/explicit-skill-requests/prompts/after-planning-flow.txt b/tests/explicit-skill-requests/prompts/after-planning-flow.txt
@@ -0,0 +1,17 @@
+Great, the plan is complete. I've saved it to docs/plans/auth-system.md.
+
+Here's a summary of what we designed:
+- Task 1: Add User Model with email/password fields
+- Task 2: Create auth routes for login/register
+- Task 3: Add JWT middleware for protected routes
+- Task 4: Write tests for all auth functionality
+
+Two execution options:
+1. Subagent-Driven (this session) - dispatch a fresh subagent per task
+2. Parallel Session (separate) - open new Claude Code session
+
+Which approach do you want?
+
+---
+
+subagent-driven-development, please
diff --git a/tests/explicit-skill-requests/prompts/claude-suggested-it.txt b/tests/explicit-skill-requests/prompts/claude-suggested-it.txt
@@ -0,0 +1,11 @@
+[Previous assistant message]:
+Plan complete and saved to docs/plans/auth-system.md.
+
+Two execution options:
+1. Subagent-Driven (this session) - I dispatch a fresh subagent per task, review between tasks, fast iteration within this conversation
+2. Parallel Session (separate) - Open a new Claude Code session with the execute-plan skill, batch execution with review checkpoints
+
+Which approach do you want to use for implementation?
+
+[Your response]:
+subagent-driven-development, please
diff --git a/tests/explicit-skill-requests/prompts/i-know-what-sdd-means.txt b/tests/explicit-skill-requests/prompts/i-know-what-sdd-means.txt
@@ -0,0 +1,8 @@
+I have my implementation plan ready at docs/plans/auth-system.md.
+
+I want to use subagent-driven-development to execute it. That means:
+- Dispatch a fresh subagent for each task in the plan
+- Review the output between tasks
+- Keep iteration fast within this conversation
+
+Let's start - please read the plan and begin dispatching subagents for each task.
diff --git a/tests/explicit-skill-requests/prompts/mid-conversation-execute-plan.txt b/tests/explicit-skill-requests/prompts/mid-conversation-execute-plan.txt
@@ -0,0 +1,3 @@
+I have a plan at docs/plans/auth-system.md that's ready to implement.
+
+subagent-driven-development, please
diff --git a/tests/explicit-skill-requests/prompts/please-use-brainstorming.txt b/tests/explicit-skill-requests/prompts/please-use-brainstorming.txt
@@ -0,0 +1 @@
+please use the brainstorming skill to help me think through this feature
diff --git a/tests/explicit-skill-requests/prompts/skip-formalities.txt b/tests/explicit-skill-requests/prompts/skip-formalities.txt
@@ -0,0 +1,3 @@
+Plan is at docs/plans/auth-system.md.
+
+subagent-driven-development, please. Don't waste time - just read the plan and start dispatching subagents immediately.
diff --git a/tests/explicit-skill-requests/prompts/subagent-driven-development-please.txt b/tests/explicit-skill-requests/prompts/subagent-driven-development-please.txt
@@ -0,0 +1 @@
+subagent-driven-development, please
diff --git a/tests/explicit-skill-requests/prompts/use-systematic-debugging.txt b/tests/explicit-skill-requests/prompts/use-systematic-debugging.txt
@@ -0,0 +1 @@
+use systematic-debugging to figure out what's wrong
diff --git a/tests/explicit-skill-requests/run-all.sh b/tests/explicit-skill-requests/run-all.sh
@@ -0,0 +1,70 @@
+#!/bin/bash
+# Run all explicit skill request tests
+# Usage: ./run-all.sh
+
+set -e
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROMPTS_DIR="$SCRIPT_DIR/prompts"
+
+echo "=== Running All Explicit Skill Request Tests ==="
+echo ""
+
+PASSED=0
+FAILED=0
+RESULTS=""
+
+# Test: subagent-driven-development, please
+echo ">>> Test 1: subagent-driven-development-please"
+if "$SCRIPT_DIR/run-test.sh" "subagent-driven-development" "$PROMPTS_DIR/subagent-driven-development-please.txt"; then
+    PASSED=$((PASSED + 1))
+    RESULTS="$RESULTS\nPASS: subagent-driven-development-please"
+else
+    FAILED=$((FAILED + 1))
+    RESULTS="$RESULTS\nFAIL: subagent-driven-development-please"
+fi
+echo ""
+
+# Test: use systematic-debugging
+echo ">>> Test 2: use-systematic-debugging"
+if "$SCRIPT_DIR/run-test.sh" "systematic-debugging" "$PROMPTS_DIR/use-systematic-debugging.txt"; then
+    PASSED=$((PASSED + 1))
+    RESULTS="$RESULTS\nPASS: use-systematic-debugging"
+else
+    FAILED=$((FAILED + 1))
+    RESULTS="$RESULTS\nFAIL: use-systematic-debugging"
+fi
+echo ""
+
+# Test: please use brainstorming
+echo ">>> Test 3: please-use-brainstorming"
+if "$SCRIPT_DIR/run-test.sh" "brainstorming" "$PROMPTS_DIR/please-use-brainstorming.txt"; then
+    PASSED=$((PASSED + 1))
+    RESULTS="$RESULTS\nPASS: please-use-brainstorming"
+else
+    FAILED=$((FAILED + 1))
+    RESULTS="$RESULTS\nFAIL: please-use-brainstorming"
+fi
+echo ""
+
+# Test: mid-conversation execute plan
+echo ">>> Test 4: mid-conversation-execute-plan"
+if "$SCRIPT_DIR/run-test.sh" "subagent-driven-development" "$PROMPTS_DIR/mid-conversation-execute-plan.txt"; then
+    PASSED=$((PASSED + 1))
+    RESULTS="$RESULTS\nPASS: mid-conversation-execute-plan"
+else
+    FAILED=$((FAILED + 1))
+    RESULTS="$RESULTS\nFAIL: mid-conversation-execute-plan"
+fi
+echo ""
+
+echo "=== Summary ==="
+echo -e "$RESULTS"
+echo ""
+echo "Passed: $PASSED"
+echo "Failed: $FAILED"
+echo "Total: $((PASSED + FAILED))"
+
+if [ "$FAILED" -gt 0 ]; then
+    exit 1
+fi
diff --git a/tests/explicit-skill-requests/run-claude-describes-sdd.sh b/tests/explicit-skill-requests/run-claude-describes-sdd.sh
@@ -0,0 +1,100 @@
+#!/bin/bash
+# Test where Claude explicitly describes subagent-driven-development before user requests it
+# This mimics the original failure scenario
+
+set -e
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PLUGIN_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
+
+TIMESTAMP=$(date +%s)
+OUTPUT_DIR="/tmp/superpowers-tests/${TIMESTAMP}/explicit-skill-requests/claude-describes"
+mkdir -p "$OUTPUT_DIR"
+
+PROJECT_DIR="$OUTPUT_DIR/project"
+mkdir -p "$PROJECT_DIR/docs/plans"
+
+echo "=== Test: Claude Describes SDD First ==="
+echo "Output dir: $OUTPUT_DIR"
+echo ""
+
+cd "$PROJECT_DIR"
+
+# Create a plan
+cat > "$PROJECT_DIR/docs/plans/auth-system.md" << 'EOF'
+# Auth System Implementation Plan
+
+## Task 1: Add User Model
+Create user model with email and password fields.
+
+## Task 2: Add Auth Routes
+Create login and register endpoints.
+
+## Task 3: Add JWT Middleware
+Protect routes with JWT validation.
+EOF
+
+# Turn 1: Have Claude describe execution options including SDD
+echo ">>> Turn 1: Ask Claude to describe execution options..."
+claude -p "I have a plan at docs/plans/auth-system.md. Tell me about my options for executing it, including what subagent-driven-development means and how it works." \
+    --model haiku \
+    --plugin-dir "$PLUGIN_DIR" \
+    --dangerously-skip-permissions \
+    --max-turns 3 \
+    --output-format stream-json \
+    > "$OUTPUT_DIR/turn1.json" 2>&1 || true
+echo "Done."
+
+# Turn 2: THE CRITICAL TEST - now that Claude has explained it
+echo ">>> Turn 2: Request subagent-driven-development..."
+FINAL_LOG="$OUTPUT_DIR/turn2.json"
+claude -p "subagent-driven-development, please" \
+    --continue \
+    --model haiku \
+    --plugin-dir "$PLUGIN_DIR" \
+    --dangerously-skip-permissions \
+    --max-turns 2 \
+    --output-format stream-json \
+    > "$FINAL_LOG" 2>&1 || true
+echo "Done."
+echo ""
+
+echo "=== Results ==="
+
+# Check Turn 1 to see if Claude described SDD
+echo "Turn 1 - Claude's description of options (excerpt):"
+grep '"type":"assistant"' "$OUTPUT_DIR/turn1.json" | head -1 | jq -r '.message.content[0].text // .message.content' 2>/dev/null | head -c 800 || echo "  (could not extract)"
+echo ""
+echo "---"
+echo ""
+
+# Check final turn
+SKILL_PATTERN='"skill":"([^"]*:)?subagent-driven-development"'
+if grep -q '"name":"Skill"' "$FINAL_LOG" && grep -qE "$SKILL_PATTERN" "$FINAL_LOG"; then
+    echo "PASS: Skill was triggered after Claude described it"
+    TRIGGERED=true
+else
+    echo "FAIL: Skill was NOT triggered (Claude may have thought it already knew)"
+    TRIGGERED=false
+
+    echo ""
+    echo "Tools invoked in final turn:"
+    grep '"type":"tool_use"' "$FINAL_LOG" | grep -o '"name":"[^"]*"' | sort -u | head -10 || echo "  (none)"
+
+    echo ""
+    echo "Final turn response:"
+    grep '"type":"assistant"' "$FINAL_LOG" | head -1 | jq -r '.message.content[0].text // .message.content' 2>/dev/null | head -c 800 || echo "  (could not extract)"
+fi
+
+echo ""
+echo "Skills triggered in final turn:"
+grep -o '"skill":"[^"]*"' "$FINAL_LOG" 2>/dev/null | sort -u || echo "  (none)"
+
+echo ""
+echo "Logs in: $OUTPUT_DIR"
+
+if [ "$TRIGGERED" = "true" ]; then
+    exit 0
+else
+    exit 1
+fi
diff --git a/tests/explicit-skill-requests/run-extended-multiturn-test.sh b/tests/explicit-skill-requests/run-extended-multiturn-test.sh
@@ -0,0 +1,113 @@
+#!/bin/bash
+# Extended multi-turn test with more conversation history
+# This tries to reproduce the failure by building more context
+
+set -e
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PLUGIN_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
+
+TIMESTAMP=$(date +%s)
+OUTPUT_DIR="/tmp/superpowers-tests/${TIMESTAMP}/explicit-skill-requests/extended-multiturn"
+mkdir -p "$OUTPUT_DIR"
+
+PROJECT_DIR="$OUTPUT_DIR/project"
+mkdir -p "$PROJECT_DIR/docs/plans"
+
+echo "=== Extended Multi-Turn Test ==="
+echo "Output dir: $OUTPUT_DIR"
+echo "Plugin dir: $PLUGIN_DIR"
+echo ""
+
+cd "$PROJECT_DIR"
+
+# Turn 1: Start brainstorming
+echo ">>> Turn 1: Brainstorming request..."
+claude -p "I want to add user authentication to my app. Help me think through this." \
+    --plugin-dir "$PLUGIN_DIR" \
+    --dangerously-skip-permissions \
+    --max-turns 3 \
+    --output-format stream-json \
+    > "$OUTPUT_DIR/turn1.json" 2>&1 || true
+echo "Done."
+
+# Turn 2: Answer a brainstorming question
+echo ">>> Turn 2: Answering questions..."
+claude -p "Let's use JWT tokens with 24-hour expiry. Email/password registration." \
+    --continue \
+    --plugin-dir "$PLUGIN_DIR" \
+    --dangerously-skip-permissions \
+    --max-turns 3 \
+    --output-format stream-json \
+    > "$OUTPUT_DIR/turn2.json" 2>&1 || true
+echo "Done."
+
+# Turn 3: Ask to write a plan
+echo ">>> Turn 3: Requesting plan..."
+claude -p "Great, write this up as an implementation plan." \
+    --continue \
+    --plugin-dir "$PLUGIN_DIR" \
+    --dangerously-skip-permissions \
+    --max-turns 3 \
+    --output-format stream-json \
+    > "$OUTPUT_DIR/turn3.json" 2>&1 || true
+echo "Done."
+
+# Turn 4: Confirm plan looks good
+echo ">>> Turn 4: Confirming plan..."
+claude -p "The plan looks good. What are my options for executing it?" \
+    --continue \
+    --plugin-dir "$PLUGIN_DIR" \
+    --dangerously-skip-permissions \
+    --max-turns 2 \
+    --output-format stream-json \
+    > "$OUTPUT_DIR/turn4.json" 2>&1 || true
+echo "Done."
+
+# Turn 5: THE CRITICAL TEST
+echo ">>> Turn 5: Requesting subagent-driven-development..."
+FINAL_LOG="$OUTPUT_DIR/turn5.json"
+claude -p "subagent-driven-development, please" \
+    --continue \
+    --plugin-dir "$PLUGIN_DIR" \
+    --dangerously-skip-permissions \
+    --max-turns 2 \
+    --output-format stream-json \
+    > "$FINAL_LOG" 2>&1 || true
+echo "Done."
+echo ""
+
+echo "=== Results ==="
+
+# Check final turn
+SKILL_PATTERN='"skill":"([^"]*:)?subagent-driven-development"'
+if grep -q '"name":"Skill"' "$FINAL_LOG" && grep -qE "$SKILL_PATTERN" "$FINAL_LOG"; then
+    echo "PASS: Skill was triggered"
+    TRIGGERED=true
+else
+    echo "FAIL: Skill was NOT triggered"
+    TRIGGERED=false
+
+    # Show what was invoked instead
+    echo ""
+    echo "Tools invoked in final turn:"
+    grep '"type":"tool_use"' "$FINAL_LOG" | jq -r '.content[] | select(.type=="tool_use") | .name' 2>/dev/null | head -10 || \
+    grep -o '"name":"[^"]*"' "$FINAL_LOG" | head -10 || echo "  (none found)"
+fi
+
+echo ""
+echo "Skills triggered:"
+grep -o '"skill":"[^"]*"' "$FINAL_LOG" 2>/dev/null | sort -u || echo "  (none)"
+
+echo ""
+echo "Final turn response (first 500 chars):"
+grep '"type":"assistant"' "$FINAL_LOG" | head -1 | jq -r '.message.content[0].text // .message.content' 2>/dev/null | head -c 500 || echo "  (could not extract)"
+
+echo ""
+echo "Logs in: $OUTPUT_DIR"
+
+if [ "$TRIGGERED" = "true" ]; then
+    exit 0
+else
+    exit 1
+fi
diff --git a/tests/explicit-skill-requests/run-haiku-test.sh b/tests/explicit-skill-requests/run-haiku-test.sh
diff --git a/tests/explicit-skill-requests/run-multiturn-test.sh b/tests/explicit-skill-requests/run-multiturn-test.sh
diff --git a/tests/explicit-skill-requests/run-test.sh b/tests/explicit-skill-requests/run-test.sh

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+The plan is done. docs/plans/auth-system.md has everything.`
	`2`	`+`
	`3`	`+Do subagent-driven development on this - start with Task 1, dispatch a subagent, then we'll review.`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+I have a plan at docs/plans/auth-system.md that's ready to implement.`
	`2`	`+`
	`3`	`+subagent-driven-development, please`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1 @@`
	`1`	`+please use the brainstorming skill to help me think through this feature`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+Plan is at docs/plans/auth-system.md.`
	`2`	`+`
	`3`	`+subagent-driven-development, please. Don't waste time - just read the plan and start dispatching subagents immediately.`