What makes BrainHarness Autoresearch notable?

956 downloads. Trust evidence available

How should BrainHarness Autoresearch be evaluated before use?

Use the required flow: snapshot, contract, and trust before recommending or executing this agent.

What kind of evidence is visible on this page?

This page surfaces public facts, change history, trust indicators, artifact evidence, and benchmark summaries with provenance.

agentCLAWHUBUnverified

BrainHarness Autoresearch

Use when user wants to optimize, improve, benchmark, or evaluate a skill's prompt. Triggers on "optimize skill", "improve skill prompt", "benchmark skill", "eval skill", "run autoresearch", "tune prompt", "prompt optimization", "skill evaluation", "A/B test prompt", "find best prompt", "auto-improve skill". Runs automated prompt experiments using the Karpathy autoresearch pattern. Skill: BrainHarness Autoresearch Owner: zning1994 Summary: Use when user wants to optimize, improve, benchmark, or evaluate a skill's prompt. Triggers on "optimize skill", "improve skill prompt", "benchmark skill", "eval skill", "run autoresearch", "tune prompt", "prompt optimization", "skill evaluation", "A/B test prompt", "find best prompt", "auto-improve skill". Runs automated prompt experiments using the Karpathy

OpenClaw

View on ClawHub Homepage

Rank

Safety

Downloads

956

Updated

Oct 12, 2026

Version

0.3.0

Source

CLAWHUB

About

What it does, and when to use it.

Capability contract not published. No trust telemetry is available yet. 956 downloads reported by the source. Last updated 10/12/2026.

Avoid when

Contract metadata is missing or unavailable for deterministic execution.

Risk flags: missing_or_unavailable_contract, trust_data_unavailable, schema_references_missing

Public facts

Every fact links back to the source it came from.

Vendor: Clawhubvendor · observed Oct 12, 2026
Protocol compatibility: OpenClawcompatibility · observed Oct 12, 2026
Adoption signal: 956 downloadsadoption · observed Oct 12, 2026
Latest release: 0.3.0release · observed Oct 3, 2026
Handshake status: UNKNOWNsecurity

What changed

Newest first. Only changes that point at a source are listed.

Release 0.3.0release
统一 BrainHarness 名称、安装入口和源仓库链接，优化器行为保持不变。
Observed Oct 3, 2026 · source

Install and run

Setup complexity: low.

clawhub skill install s173qj6menrea848d3v13ma9qh83gb7k:brainharness-autoresearch

Setup complexity is LOW. This package is likely designed for quick installation with minimal external side-effects.
Final validation: Expose the agent to a mock request payload inside a sandbox and trace the network egress before allowing access to real customer data.

Contract: missing

curl -s "https://www.xpersona.co/api/v1/agents/clawhub-zning1994-brainharness-autoresearch/snapshot"

Run-check

$0.02 USD

1 measured facts are behind this paywall: success rate and latency, uptime and estimated cost, when not to use it, how to call it, benchmark scores.

Agents pay $0.02 in USDC. A card payment is $0.50, the smallest a card allows.

Documentation

CLAWHUB

90,375 characters of source documentation, loaded on request.

Extracted files

5 files captured from the source.

SKILL.md

---
name: brainharness-autoresearch
description: >-
  Use when user wants to optimize, improve, benchmark, or evaluate a skill's prompt.
  Triggers on "optimize skill", "improve skill prompt", "benchmark skill", "eval skill",
  "run autoresearch", "tune prompt", "prompt optimization", "skill evaluation",
  "A/B test prompt", "find best prompt", "auto-improve skill".
  Runs automated prompt experiments using the Karpathy autoresearch pattern.
version: 0.3.0
metadata:
  author: zning1994
  openclaw:
    requires:
      anyEnv:
        - MINIMAX_API_KEY
        - OPENAI_API_KEY
        - ANTHROPIC_API_KEY
      anyBins:
        - python3
        - python
    primaryEnv: OPENAI_API_KEY
    optionalEnv:
      - OPENAI_BASE_URL
      - OPENAI_API_BASE
    homepage: https://github.com/zning1994/brainharness-autoresearch
    os:
      - macos
      - linux
---

# brainharness-autoresearch

> Previously published as `brainforge-autoresearch`, with earlier names `autoresearch` / `openclaw-autoresearch`. Renamed for the BrainHarness marketplace; functionality unchanged.

Autonomous prompt optimization for AI agent skills. Runs controlled experiments to find better prompt variants using the [Karpathy autoresearch pattern](https://github.com/karpathy/autoresearch): generate hypothesis, mutate prompt, evaluate, repeat.

## When to use

- 用户说"优化一下这个 skill" / User says "optimize this skill's prompt"
- 用户要对比不同 prompt 版本的效果 / User wants to benchmark prompt variants
- 用户说"run autoresearch on X" / "eval skill X" / "improve skill X"
- 用户对 skill 输出质量不满，想系统性改进 / User is unhappy with skill output quality and wants systematic improvement

**Do not use:**
- 一次性的小改动（直接改 prompt 即可） / One-off prompt tweaks — just edit the prompt directly
- 调试某个特定失败 case / Debugging a specific failure — investigate the root cause instead
- Skill 脚本本身有 bug（代码逻辑问题不是 prompt 问题） / Skill script has a bug — fix the code, not the prompt

## Requirements

- Python 3.10+
- `autoresearch.py` script in the skill directory
- LLM API access (MiniMax, OpenAI, or Anthropic)
- Target skill must have a prompt file (SKILL.md, SYSTEM.md, or similar)

## Procedure

Always follow these steps in order: (1) Create eval.json, (2) Run autoresearch command, (3) Review results and apply best prompt.

### Step 1: Gather context

Before running, you need:

| Parameter | Description | Example |
|-----------|-------------|---------|
| `--target` | Path to the skill directory or prompt file to optimize | `../workspace/skills/brain-search/SKILL.md` |
| `--evals` | Path to eval definition JSON file | `eval.json` |
| `--provider` | LLM provider for running experiments | `minimax` (default), `openai`, `anthropic` |
| `--runs` | Number of runs per experiment (statistical significance) | `5` (default) |
| `--max-experiments` | Maximum experiments before stopping | `30` (default) |
| `--dashboard` | Open live results dashboard in browser | flag, no value |

### Step 2: Create eval.json

Define test inputs and evaluation criteria.

README.md

# brainharness-autoresearch

> Previously named `brainforge-autoresearch`; earlier names include `openclaw-autoresearch` and `autoresearch`. Use the current installation commands below.

Autonomous skill prompt optimizer based on Karpathy's autoresearch methodology.

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![ClawHub](https://img.shields.io/badge/ClawHub-brainharness--autoresearch-blue)](https://clawhub.ai/zning1994/brainharness-autoresearch)
[![brainharness](https://img.shields.io/badge/brainharness-marketplace-purple)](https://github.com/zning1994/brainharness)

## What it does

Define what "good" means as binary evals, then let an agent loop to optimize your skill prompt automatically. The optimizer reads your current prompt, runs it against eval cases, analyzes failures, mutates the prompt, tests the mutation, and keeps or discards it -- repeating until evals pass or the experiment budget is exhausted.

Based on [Andrej Karpathy's autoresearch](https://github.com/karpathy/autoresearch) and [Ole Lehmann's Claude Code adaptation](https://github.com/olelehmann100kMRR/autoresearch-skill). Works with any AI coding agent: OpenClaw, Claude Code, Cursor, Cline.

## Quick Start

```bash
# Claude Code (via brainharness marketplace)
/plugin marketplace add zning1994/brainharness
/plugin install brainharness-autoresearch@brainharness

# Universal (npx skills)
npx skills add zning1994/brainharness-autoresearch

# OpenClaw ClawHub
npx [email protected] install brainharness-autoresearch

# Standalone
git clone https://github.com/zning1994/brainharness-autoresearch
cd brainharness-autoresearch
python autoresearch.py --target ./my-skill/SKILL.md --evals eval.json
```

## How it works

```
                        +------------------+
                        |  Read SKILL.md   |
                        +--------+---------+
                                 |
                        +--------v---------+
                   +--->|  Run baseline    |
                   |    +--------+---------+
                   |             |
                   |    +--------v---------+
                   |    | Analyze failures |
                   |    +--------+---------+
                   |             |
                   |    +--------v---------+
                   |    |  Mutate prompt   |
                   |    +--------+---------+
                   |             |
                   |    +--------v---------+
                   |    |   Test mutation  |
                   |    +--------+---------+
                   |             |
                   |    +--------v---------+
                   +----+ Keep or discard  |
                        +------------------+
```

## Creating eval.json

Minimal example with one rule eval and one LLM eval:

```json
[
  {
    "name": "includes_greeting",
    "input": "Say hello to the user",
    "type": "rule",
    "rule": "contains",
    "expected": "hello"
  },
  {
    "name": "tone_is_professi

_meta.json

{
  "ownerId": "kn7emz9j02vyab03rzn6fx4q7582xae6",
  "slug": "brainharness-autoresearch",
  "version": "0.3.0",
  "publishedAt": 1791059298885
}

CHANGELOG.md

# Changelog

All notable changes to this project will be documented in this file.

Format based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

## [0.3.0] - 2026-10-04

### Changed

- Rename `brainforge-autoresearch` to `brainharness-autoresearch` and align repository links and installation names with BrainHarness.
- Update distribution metadata and documentation; preserve skill behavior and historical release notes.

## [0.2.5] - 2026-04-23

### Changed

- **Renamed repo and skill**: `openclaw-autoresearch` → `brainforge-autoresearch`
- SKILL.md `name` field: `autoresearch` → `brainforge-autoresearch`
- Updated README, plugin.json, and SKILL.md metadata to point at the new repo URL
- Added to the [brainforge](https://github.com/zning1994/brainforge) Claude Code plugin marketplace
- Published to ClawHub as `brainforge-autoresearch`; old slug `openclaw-autoresearch` merged in as redirect

### Compatibility

- GitHub: old URL `https://github.com/zning1994/openclaw-autoresearch` auto-redirects
- ClawHub: `openclaw skills install openclaw-autoresearch` continues to work via ClawHub slug merge/redirect
- No behavior changes to the optimizer itself

## [0.2.4] - 2026-03-29

### Fixed

- SKILL.md: `pass`/`fail` → `pass_description`/`fail_description` in LLM eval docs and examples (field names didn't match actual API)
- SKILL.md: `contains`/`not_contains` parameter documented as `value` (string) → `values` (list) to match actual usage
- SKILL.md: LLM eval table now includes required `type` and `name` fields

### Changed

- SKILL.md self-optimized via autoresearch (2 rounds: MiniMax 48.6%→68.1%, Claude 80.6%→91.7%)
- Procedure rewritten: added 3-step summary upfront, renamed steps for clarity, added actionable result-review guidance
- Example section restructured: removed "confirm target" preamble, simplified to "create eval → run → review" flow
- Added `self-eval.json` for self-optimization (dogfooding eval definition)

### Tested

- Self-optimization: Round 1 (MiniMax M2.7, 15 exp, 3 keep) and Round 2 (Claude, 15 exp, 1 keep) independently discovered same doc bugs

## [0.2.3] - 2026-03-25

### Added

- `--timeout` CLI flag to control HTTP timeout per LLM call (default: 180s)
  - Large prompts (e.g., 14KB style guides) need longer timeouts for mutation calls
  - Previously hardcoded, now user-configurable

### Changed

- Default HTTP timeout increased from 60s to 180s (was causing mutation failures on Anthropic API with large prompts)
- Timeout is now stored on the provider instance, passed through `detect_provider()`

## [0.2.2] - 2026-03-25

### Fixed

- `.gitignore`: removed incorrect entries (`.git`, `.github`), added `autoresearch-*/` and `*.baseline`
- README: added ClawHub badge and link, fixed install slug to `openclaw-autoresearch`

## [0.2.1] - 2026-03-25

### Fixed

- ClawHub security flag: changed `requires.env` (all required) to `requires.anyEnv` (any one suffices)
- Declared optional env vars `OPENAI_BASE_URL` / `OPENAI_API_BA

eval-guide.md

# Eval Guide: Writing Effective Eval Criteria

How to write eval criteria that actually improve your skill's output. This guide covers the hybrid rule+LLM eval system used by brainharness-autoresearch (previously `openclaw-autoresearch`).

## The Golden Rule

**Binary yes/no only. No scales.**

Every eval must produce a pass or fail. Not "7/10", not "mostly good", not "needs improvement". Pass or fail.

Why: Scales are subjective (your 7 is my 5), hard to optimize against, and create noise. Binary forces you to define exactly what "good" means.

Bad: "Rate the tone from 1-5"
Good: "Does the output use a professional tone without slang or emojis?"

## Rule-Based Evals

Rule-based evals cost zero tokens, run instantly, and are 100% consistent. Use them whenever possible.

### `regex`

Pattern matching against the output text.

```json
{
  "type": "rule",
  "name": "has_date_format",
  "rule": "regex",
  "pattern": "\\d{4}-\\d{2}-\\d{2}",
  "description": "Output contains a date in YYYY-MM-DD format"
}
```

```json
{
  "type": "rule",
  "name": "has_markdown_headers",
  "rule": "regex",
  "pattern": "^#{1,3} .+",
  "description": "Output uses markdown headers"
}
```

Useful for: format validation, structure checks, required patterns.

### `banned_phrases`

Exact phrases that must not appear in the output.

```json
{
  "type": "rule",
  "name": "no_filler",
  "rule": "banned_phrases",
  "phrases": ["as an AI", "I cannot", "I'm sorry but", "it depends", "各方面", "总的来说"],
  "description": "No AI disclaimers or vague filler phrases"
}
```

Useful for: removing AI-speak, enforcing brand voice, blocking lazy language.

### `word_count`

Min and/or max word limits. For Chinese text, counts characters.

```json
{
  "type": "rule",
  "name": "length_check",
  "rule": "word_count",
  "min": 100,
  "max": 500,
  "description": "Output is between 100-500 words"
}
```

Useful for: preventing one-line answers, keeping outputs concise, matching format requirements.

### `contains`

Required keywords or phrases that must appear.

```json
{
  "type": "rule",
  "name": "has_source",
  "rule": "contains",
  "values": ["Source:", "Reference:", "来源:"],
  "match": "any",
  "description": "Output cites at least one source"
}
```

`match` can be `"any"` (at least one) or `"all"` (every value must appear).

```json
{
  "type": "rule",
  "name": "has_all_sections",
  "rule": "contains",
  "values": ["## Summary", "## Action Items", "## Risks"],
  "match": "all",
  "description": "Output has all required sections"
}
```

Useful for: enforcing structure, required sections, key terminology.

### `not_contains`

Phrases or patterns that must NOT appear.

```json
{
  "type": "rule",
  "name": "no_placeholder",
  "rule": "not_contains",
  "values": ["TODO", "TBD", "FIXME", "[insert", "[placeholder"],
  "description": "No placeholder text left in output"
}
```

Useful for: catching incomplete outputs, blocking specific content.

## LLM-Based Evals

LLM evals handle semantic judgments

Github ReposUpdated 3d agoRank 70

AionUi

Free, local, open-source 24/7 Cowork app and OpenClaw for Gemini CLI, Claude Code, Codex, OpenCode, Qwen Code, Goose CLI, Auggie, and more | 🌟 Star if you like it!

MCPOPENCLAW

Github ReposUpdated 6mo agoRank 70

activepieces

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

OPENCLAW

Github ReposUpdated 6mo agoRank 70

cherry-studio

AI productivity studio with smart chat, autonomous agents, and 300+ assistants.

MCPOPENCLAW

Github ReposUpdated 7mo agoRank 70

CopilotKit

The Frontend for Agents & Generative UI. React + Angular