---

name: skill-studio description: "Comprehensive skill evaluation and debugging framework for testing agent skills. Use when users need to (1) evaluate a skill's recall rate (how often it triggers correctly), (2) test a skill's accuracy against expected outputs, (3) analyze skill performance with various prompts, or (4) generate improvement recommendations for existing skills. Requires test cases with standard answers."

Skill Studio

A systematic framework for evaluating and debugging agent skills through automated testing of recall rates and output accuracy.

Overview

Skill Studio provides tools and workflows to:

• Test if a skill triggers correctly (recall rate)
• Verify skill outputs match expected results (accuracy rate)
• Evaluate code outputs with specialized metrics (syntax, structure, functionality, quality)
• Generate actionable improvement recommendations
• Track performance across iterations

Core Workflow

Step 1: Collect Test Requirements

Gather from the user:

• Target skill name: The skill being evaluated (e.g., "pptx", "docx", "custom-skill")
• Test cases file: Path to test prompts (or help create one)
• Standard answers file: Path to expected outputs (or help create one)
• Test parameters: Number of prompt variations, testing approach

Step 2: Prepare Test Cases

If test cases don't exist, help create test_cases.json using the format in references/test_format.md.

Key elements:

• Diverse prompts that should trigger the skill
• Negative cases (prompts that should NOT trigger)
• Category labels for analysis
• Expected behaviors

Step 3: Run Recall Testing

Execute recall rate evaluation:

python3 scripts/test_recall.py \
  --skill-name <skill-name> \
  --test-cases <path-to-test-cases.json> \
  --variations 5 \
  --output outputs/recall_results.json

What it does:

• Generates prompt variations for each test case
• Tests if the skill triggers with each variation
• Calculates recall rates overall and by category
• Identifies false positives and false negatives

Key metrics:

• Overall recall rate: % of prompts that correctly triggered
• Per-category recall: Performance by test case type
• False positive rate: Incorrect triggering

Step 4: Run Accuracy Testing

Execute output accuracy evaluation:

python3 scripts/test_accuracy.py \
  --skill-name <skill-name> \
  --test-cases <path-to-test-cases.json> \
  --standard-answers <path-to-standard-answers.json> \
  --workspace <path-to-workspace-directory> \
  --artifacts-dir test_artifacts \
  --output outputs/accuracy_results.json

What it does:

• Runs the skill with test prompts
• Captures file changes (for coding agent skills)
• Compares outputs to standard answers
• Evaluates both exact matches and semantic similarity
• Saves before/after states and diffs
• Categorizes errors by type

Key parameters:

• --workspace: Optional. Path to workspace directory for capturing code changes
• --artifacts-dir: Optional. Directory to save test artifacts (default: test_artifacts)

Key metrics:

• Exact match accuracy: % of perfect matches
• Semantic similarity: Average similarity score (0-1)
• Error categories: Common failure patterns

Artifacts saved (when --workspace is provided):

For each test case, the following artifacts are saved in test_artifacts/<test_id>/:

• metadata.json: Test case information
• before_state.json: File states before skill execution
• after_state.json: File states after skill execution
• changes.json: Summary of added/modified/deleted files
• diffs/<file>.diff: Unified diffs for each modified file
• full_output.json: Complete skill response

This allows you to:

• Review exact code changes made by the skill
• Compare actual changes to expected changes
• Debug why tests failed
• Track how the skill evolves over iterations

For code outputs:

• Syntax validity: Does code parse correctly?
• Structure match: Has expected functions/classes/imports?
• Functionality: Passes test cases?
• Code quality: Comments, docstrings, error handling, type hints

Step 5: Analyze and Generate Recommendations

Run the analysis to get improvement suggestions:

python3 scripts/analyze_results.py \
  --recall-results outputs/recall_results.json \
  --accuracy-results outputs/accuracy_results.json \
  --skill-path <path-to-skill-folder> \
  --output outputs/recommendations.md

Output includes:

• Performance summary with visual indicators
• Specific issues identified (low-performing categories)
• Concrete recommendations for SKILL.md improvements
• Suggested trigger phrases for description
• Priority-ranked action items

Step 6: Implement Improvements

Based on recommendations:

1. Review the generated recommendations document
2. Update the skill's SKILL.md frontmatter description
3. Enhance instructions in SKILL.md body
4. Add clarifying examples or decision trees
5. Re-run tests to validate improvements

Iterate until target performance is achieved.

Interpreting Results

Recall Rate Thresholds

• >90%: Excellent - skill triggers reliably
• 70-90%: Good - minor description improvements helpful
• 50-70%: Fair - description needs refinement
• <50%: Poor - major triggering issues, likely unclear description

Accuracy Rate Thresholds

• >85%: Excellent - outputs are high quality
• 70-85%: Good - minor instruction improvements helpful
• 55-70%: Fair - instructions need clarification
• <55%: Poor - major workflow or instruction issues

Common Issues and Solutions

Low Recall Rate:

• Description is too vague or generic
• Missing key trigger keywords
• Overlaps with other skills' descriptions
• Solution: Add specific file types, actions, or scenarios to description

High False Positive Rate:

• Description is too broad
• Includes common generic terms
• Solution: Be more specific about exact use cases

Low Accuracy Rate:

• Instructions are unclear or incomplete
• Missing critical steps in workflow
• Examples don't match user expectations
• Solution: Add decision trees, more examples, clearer step-by-step guidance

Variable Performance by Category:

• Skill handles some use cases well but not others
• Solution: Add specific guidance for underperforming categories

Best Practices

Test Case Design

• Include 20-30 diverse test cases minimum
• Cover all major use cases the skill should handle
• Add negative cases (should NOT trigger) at ~20% ratio
• Use realistic, varied phrasing
• Label test cases by category for granular analysis

Standard Answers

• Define clear quality criteria
• Use semantic similarity for flexible matching when appropriate
• Include both structure and content expectations
• Specify file formats and key elements

Iterative Testing

• Test after each modification to track progress
• Focus on lowest-performing categories first
• Keep a log of changes and their impact
• Aim for incremental improvements (5-10% gains per iteration)

Skill Description Optimization

• Include specific file extensions (e.g., ".docx", ".pptx")
• List concrete actions (e.g., "creating", "editing", "analyzing")
• Mention key scenarios (e.g., "when user uploads", "when user requests")
• Use numbered lists for multiple trigger conditions
• Avoid generic terms that apply to many skills

Resources

scripts/

• test_recall.py: Measures skill triggering consistency
• test_accuracy.py: Evaluates output quality
• analyze_results.py: Generates improvement recommendations
• generate_variations.py: Creates prompt variations for testing

references/

• test_format.md: Detailed test case format specification
• answer_format.md: Standard answer format specification
• code_evaluation.md: Code output evaluation format and best practices
• metrics_explained.md: Deep dive into evaluation metrics