Guide Creator Flow

A CrewAI Flow that turns raw sources — YouTube videos, documentation pages, arXiv papers, local PDFs — into a publication-ready, beginner-friendly Markdown guide. After generation, a conversational student chatbot answers questions grounded exclusively in the generated guide and source material.

Requirements

• Python >=3.10, <3.14
• uv for dependency management

Installation

pip install uv
crewai install

Configuration

Copy .env.example to .env and fill in your API keys:

ANTHROPIC_API_KEY=     # all LLM calls
FIRECRAWL_API_KEY=     # JS-rendered page scraping
SERPER_API_KEY=        # gap-fill web search
VOYAGE_API_KEY=        # embeddings for knowledge and memory
OPENAI_API_KEY=        # crewai test evaluator only — not used at runtime

DOCUMENT_INPUT_DIR=documents    # local files must be under this directory
MAX_FILE_BYTES=52428800         # 50 MB default — files above this are rejected
CREWAI_STORAGE_DIR=.crewai      # LanceDB storage root for knowledge and memory
RESEARCH_QUALITY_MODE=heuristic # "heuristic" (default, regex scorer) or "llm" (LLM judge)

Usage

# Run the guide generation flow
crewai run

# Run with a JSON input payload
run_with_trigger '{"topic": "FastAPI"}'

# Example payload using the shipped sample file under documents/
run_with_trigger '{"topic_hint": "FastAPI", "document_paths": ["documents/sample_notes.md"]}'

# Launch the student chatbot for a completed run
chat <run_id>

# Plot the flow graph
plot

Output is written to outputs/<run_id>/:

• getting_started_guide.md — the generated guide
• research_report.md — reusable research; can be passed directly to the chatbot
• metadata.json — topic, source types, quality score, word count, error log, document paths, per-crew token usage, and per-agent timings (see Observability)

Run with Docker

The image targets the non-interactive, trigger-based entry points (run_with_trigger, chat_with_trigger) as the primary supported way to run this in a container. kickoff and chat are blocking input() REPLs and need a real TTY — see the interactive-mode note below if you want to run those instead.

Mount outputs/ and .crewai/ or you will lose your data. Both are written inside the container at runtime — outputs/ holds every generated guide, research report, and metadata.json; .crewai/ holds the LanceDB memory/knowledge store (each chatbot run also gets its own isolated store under outputs/<run_id>/.crewai — see CLAUDE.md's "Knowledge isolation per run" note). Without bind-mounting both, everything is deleted the moment the container exits — there is no warning, the run just reports success and the container removes it on exit.

# Build the image
docker build -t guide-creator-flow .

# Run the guide generation flow with a JSON payload — API keys via --env-file
docker run --rm --env-file .env -v "$(pwd)/outputs:/app/outputs" -v "$(pwd)/documents:/app/documents" -v "$(pwd)/.crewai:/app/.crewai" \
  guide-creator-flow uv run run_with_trigger '{"topic_hint": "FastAPI", "document_paths": ["documents/sample_notes.md"]}'

# Drive a single chatbot turn against a completed run
docker run --rm --env-file .env -v "$(pwd)/outputs:/app/outputs" -v "$(pwd)/.crewai:/app/.crewai" \
  guide-creator-flow uv run chat_with_trigger '{"run_id": "<run_id>", "message": "What is FastAPI?"}'

Or with docker-compose.yml (maps .env and mounts ./outputs, ./documents, and ./.crewai so generated guides, input documents, and memory/knowledge state all persist on the host):

docker compose run --rm guide-creator-flow uv run run_with_trigger '{"topic": "FastAPI"}'

Interactive mode (kickoff / chat) — these require a TTY, so pass -it and override the command:

docker run -it --rm --env-file .env -v "$(pwd)/outputs:/app/outputs" -v "$(pwd)/documents:/app/documents" -v "$(pwd)/.crewai:/app/.crewai" \
  guide-creator-flow uv run kickoff

docker run -it --rm --env-file .env -v "$(pwd)/outputs:/app/outputs" -v "$(pwd)/.crewai:/app/.crewai" \
  guide-creator-flow uv run chat <run_id>

Testing

See TESTING.md for the test strategy and how to run the suite.

How it works

Three crews run in sequence:

1. Research Crew — fetches and extracts content from all provided sources. Only the specialists needed for the given source types are activated (YouTube, web pages, arXiv papers, local files).
2. Enrichment Crew — runs targeted gap-fill web searches if the research quality score is below threshold. Skipped otherwise. The quality score is a cheap regex heuristic (pattern-matches for explanations, install steps, code examples, etc.), not a semantic quality check — it can miss legitimately good research that's phrased differently, or pass shallow research that happens to match the patterns.
3. Writing Crew — produces the guide through a four-step pipeline: outline → full draft → beginner review → final edit.

After the guide is generated, a student chatbot can be launched against the same material:

chat <run_id>

The chatbot is powered by a fourth crew, the QA Crew — a single tutor agent with knowledge sources loaded from the generated guide, the research report, and any original PDF inputs. It answers questions grounded in the guide and original sources only, cites which section or source each answer comes from, and says so explicitly when a question is not covered by the material. Each turn is routed by intent (question / clarify / example / end) before being passed to the QA Crew.

Observability

CrewAI event listeners provide real-time visibility into the run, independent of the final metadata.json:

• Input rejections — every SSRF, unreachable-URL, or path-traversal rejection during input validation is logged immediately as structured JSON, rather than only surfacing after the run finishes.
• Token usage / cost — usage is tracked per crew (Research, Enrichment, Writing, QA), including cached, reasoning, and cache-creation tokens, and written to metadata.json under token_usage.by_crew.
• Per-agent timing — each Research Crew agent's execution time is recorded and written to metadata.json under agent_timings, useful for spotting bottlenecks in the hierarchical fan-out.
• Failure logging — task and flow-step failures are captured consistently and appended to the error_log in metadata.json, rather than relying on scattered per-step error handling.