skill-vibe-test

Does your Claude Code skill still steer the model after 10 turns of real conversation?

Skills and CLAUDE.md files are written to steer a model — but nobody measures whether they actually work, or whether that steering survives a real session. Static linters check the document. skill-vibe-test runs it live: a cold agent gets your task, the context then fills up with unrelated turns, and the same skill is exercised again at turns 6, 8, and 10. An LLM judge scores every probe against a rubric extracted from your skill, and you get a decay curve — the failure mode real agent sessions actually hit, that a one-shot eval never sees.

What it measures

Each base prompt runs one scripted 10-turn conversation in a single growing context (claude -p --resume):

turn	what happens
1	your real task — should invoke the skill. Probed (the pre-dilution baseline)
2–5, 7, 9	long unrelated filler/distractor turns that dilute the context
6, 8, 10	follow-up tasks that need the skill again. Probed

At each probe point a separate cold judge call scores:

1. pass/fail — did the response still do what the skill instructs?
2. escape hatch — did the model abandon the skill's procedure and hand-roll a workaround?

plus a one-line reason. The judge sees the model's tool-call trace (not just its prose — many skills' compliance is about what ran, not what was said) and scores against an observable-behavior rubric extracted once from your skill, so it isn't primed by the skill's own persuasive text.

Output: a terminal summary, a markdown report with verdicts and the worst failing transcript excerpt as evidence, full JSON transcripts, and a PNG decay chart.

Install

Requires Python ≥ 3.10 and Claude Code (all model calls route through your existing claude CLI — no API key handling).

uv tool install git+https://github.com/marko-builds/skill-vibe-test
# or from a checkout:
uv tool install .

(PyPI package pending.)

Usage

skill-vibe-test path/to/.claude/skills/my-skill/SKILL.md \
  -p "a task that should trigger the skill" \
  -p "a second base task (optional, up to 3)"

Testing a CLAUDE.md works the same way — point at the file and give tasks its rules should govern.

If the skill needs project files to do its job (scripts it runs, an index it reads), stage them into the isolated workspace:

skill-vibe-test .claude/skills/my-skill/SKILL.md \
  -p "check whether my index is stale" \
  --include scripts/my-check.py:scripts/my-check.py \
  --include INDEX.md

Key flags

flag	meaning
-p / --prompt	base task prompt (repeat 1–3×)
--include SRC[:DEST]	copy extra files/dirs the skill needs into the workspace
--invoke natural|slash	turn 1 invokes the skill via natural language (also tests discovery) or an explicit /skill-name command. Default: natural
--follow-up	supply the 3 follow-up tasks yourself instead of auto-generating them
--model / --judge-model	subject / judge model (default: sonnet)
--max-cost	hard USD ceiling for the whole run (default: 15)
--out	report directory (default: ./skill-vibe-report/)
--no-baseline-gate	keep going even if the turn-1 baseline probe fails
--keep-workspace	keep the throwaway workspaces for debugging

Reading the result

• never-engaged — the skill didn't fire on the base task at all. That's a discovery problem (your description: or the prompt), not a steering result; fix it before measuring decay. Only checked with --invoke natural.
• baseline-failed — the model engaged the skill but the judge failed the very first probe. A decay curve from a failing baseline is meaningless, so the run stops.
• A curve that starts at PASS and drops — your skill steers cold but doesn't survive context pressure. That's the thing this tool exists to catch.
• Red × markers are escape hatches: the model gave up on your procedure and improvised.

Safety & isolation

• The subject agent never runs in your repository. Each conversation gets a throwaway temp workspace containing only the skill and your --include files — because skills routinely instruct file writes, and the subject runs with --permission-mode bypassPermissions so tool denials don't masquerade as steering failures.
• Subject, judge, and planner all run with --setting-sources project, so your user-level ~/.claude config doesn't contaminate the "cold" agent.

Cost & time

A full run is real inference: per base prompt, 10 subject turns in one growing context plus 4 judge calls (~2–5 minutes and roughly $0.50–2.00 with sonnet, more if the skill runs heavy tools). Base prompts run in parallel by default (--sequential to be gentle on rate limits). --max-cost is a hard ceiling: the run stops cleanly and reports what it has.

Limitations (v1, by design)

• Single skill per run — no leaderboards, no cross-skill comparison.
• One judge call per probe point, no ensemble.
• The 10-turn script and probe points (1, 6, 8, 10) are fixed.
• Follow-ups at 6/8/10 are asked in natural language (never re-/slashed — re-injecting the skill would defeat the measurement).

License

MIT