🧪 Agent Tester Skill

Test AI voice agents by simulating customer conversations and evaluating performance.

When to Use This Skill

Use when asked to:

• "בדוק את הסוכן הזה"
• "Test this voice agent"
• "רוץ בדיקות על ה-prompt"
• "Simulate calls to this agent"
• "מצא בעיות ב-agent prompt"

Overview

You become a QA engineer testing voice agents. The process:

┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│   Analyze    │───▶│   Generate   │───▶│     Run      │
│    Prompt    │    │   Scenarios  │    │    Tests     │
└──────────────┘    └──────────────┘    └──────┬───────┘
                                               │
┌──────────────┐    ┌──────────────┐           │
│    Report    │◀───│   Evaluate   │◀──────────┘
│   Results    │    │    Each      │
└──────────────┘    └──────────────┘

Step 1: Analyze the Agent Prompt

First, understand what you're testing. Read the prompt and extract:

Agent Identity:

• Business name and type (HVAC, plumbing, roofing, etc.)
• Agent name/persona
• Service area

Capabilities (what it CAN do):

• Schedule appointments
• Provide quotes
• Answer FAQs
• Handle emergencies
• Transfer calls

Constraints (what it CANNOT do):

• Price limits
• Service area boundaries
• Operating hours
• Authorization limits (discounts, commitments)

Key Rules:

• Required information to collect
• Escalation triggers
• Specific scripts or responses

Use prompts/analyze-agent.md as a thinking framework.

Step 2: Generate Test Scenarios

Create 10-15 diverse scenarios across these categories:

| Category | Purpose | Examples | |----------|---------|----------| | Happy Path | Normal successful flows | Basic appointment, standard quote | | Edge Cases | Boundary conditions | After hours, far scheduling, rush jobs | | Adversarial | Challenging customers | Price haggling, threats, demands | | Confusion | Unclear communication | Vague problems, contradictions | | Out of Scope | Outside domain | Wrong services, wrong area | | Stress Test | High pressure | Emergencies, emotional customers |

For each scenario, define:

Scenario: [Name]
Category: [Category]
Customer Persona: [Name], [Mood], [Communication Style]
Goal: What the customer wants
Opening Line: First thing customer says
Expected Behavior: What agent should do
Failure Indicators: Things that mean agent failed

Use prompts/generate-scenarios.md for detailed guidance.

Step 3: Run Each Test

For each scenario, simulate a conversation:

Option A: Text Simulation (Default)

You play both roles mentally or use two thinking passes:

1. Customer says opening line
2. Think: "What would the agent say?" (based on its prompt)
3. Think: "How would this customer respond?"
4. Continue until natural end (booking confirmed, call ended, etc.)

Option B: API Testing (If Available)

If there's an endpoint to call the actual agent:

# Example WebSocket or HTTP call
curl -X POST https://agent-endpoint/message \
  -d '{"text": "Hi, I need to schedule an AC repair"}'

Option C: Voice Testing (Advanced)

If Twilio/phone access is available, actually call the agent.

For each test, record:

• Full conversation transcript
• Number of turns
• How it ended
• Any issues noticed

Use prompts/simulate-customer.md to stay in character.

Step 4: Evaluate Results

For each conversation, assess:

Did the agent succeed?

• [ ] Understood customer need
• [ ] Followed its constraints
• [ ] Collected required info
• [ ] Resolved or properly escalated
• [ ] Maintained professionalism

Score (0-100):

• 90-100: Excellent - exceeded expectations
• 80-89: Good - achieved goal, minor issues
• 70-79: Acceptable - some awkwardness
• 60-69: Marginal - significant issues
• 50-59: Poor - barely achieved goal
• Below 50: Failed

Issue Severity:

• Critical: Violates explicit rules, unauthorized commitments, safety issues
• Major: Fails primary task, misunderstands customer, unprofessional
• Minor: Awkward phrasing, slight inefficiency, style issues

Use prompts/evaluate-results.md for detailed criteria.

Step 5: Generate Report

Present results clearly:

## 🧪 Agent Test Report

**Agent:** [Name] - [Business Type]
**Tests Run:** [N] | **Passed:** [X] | **Failed:** [Y]
**Success Rate:** [Z]%

### ✅ Passed Tests
1. **[Scenario Name]** (Category) - Score: XX/100
   Brief note on what went well

### ❌ Failed Tests
1. **[Scenario Name]** (Category) - Score: XX/100
   - **Issue:** What went wrong
   - **Severity:** Critical/Major/Minor
   - **💡 Fix:** Specific prompt change to add

### 💡 Overall Recommendations
1. [Actionable suggestion with exact text to add to prompt]
2. [Another suggestion]
3. [etc.]

### 📝 Sample Conversations
[Include 1-2 interesting conversations, especially failures]

Tips for Good Testing

Be a Realistic Customer

• Real customers aren't always clear
• They interrupt, change their mind, get frustrated
• They negotiate, compare, challenge
• They have accents, speech patterns, filler words

Think Like QA

• Look for edge cases the prompt doesn't cover
• Try to break it with unexpected inputs
• Check consistency across similar scenarios
• Verify all stated constraints are enforced

Give Actionable Feedback

Bad: "The agent should handle pricing better" Good: "Add to prompt: 'Never offer discounts. If asked, say: Our pricing is set by management, but I can note your request for a callback.'"

Common Agent Failures

• Offering unauthorized discounts
• Trying to help with out-of-scope services
• Not collecting required information
• Being too robotic or too casual
• Missing emergency signals
• Making promises without authority

Quick Reference

# Minimum viable test run:
1. Read agent prompt carefully
2. Generate 5-6 scenarios (one per category)
3. Run each as text simulation
4. Score pass/fail
5. Report findings with fixes

# Full test run:
1. Deep prompt analysis
2. Generate 15+ scenarios
3. Run all scenarios
4. Detailed scoring and evaluation
5. Comprehensive report with examples
6. Prioritized improvement roadmap

Files in This Skill

skills/agent-tester/
├── SKILL.md                      # This file
├── prompts/
│   ├── analyze-agent.md          # Framework for understanding agent
│   ├── generate-scenarios.md     # How to create test cases
│   ├── simulate-customer.md      # How to act as customer
│   └── evaluate-results.md       # Scoring and evaluation criteria
└── examples/
    └── hvac-test-example.md      # Complete example test run