This project is a complete redesign of the original Video Extract tool.

Original implementation: https://github.com/cibis/video_extract

Video Extract Agents

A prompt-driven video extraction platform. Upload a video, describe what you want in plain English, and AI agents extract and compile the relevant segments into a highlight reel.

Example: "Extract all kitesurfing jumps from this video and compile them into a highlight reel."

Why This Exists

This project exists because of two problems.

The first: I'm mildly obsessed with agentic AI, the idea that you give a system a goal in plain English and a crew of AI agents figures out how to get there. The second: I have hours of kitesurfing footage and zero patience for scrubbing through it frame by frame.

The obvious solution was to build an enterprise-grade, cloud-native, multi-agent video extraction platform.

So here we are: a full Azure microservices stack, CrewAI orchestration, MCP tool servers, and FFmpeg keyframe pipelines, all so I can type "find the jumps" and go back to the beach.

Table of Contents

• Overview
• How It Works
• Tech Stack
• Quick Start (Local Dev)
• Running Tests
• Documentation
• External Agents
• Repository Structure

Overview

The platform combines agentic AI orchestration (CrewAI + Claude), MCP tool servers over SSE transport, and a cloud-native Azure microservices architecture to enable natural-language-driven video processing at scale.

Home — session active with completed job history and chat

Session History — completed and failed jobs with output files

Key capabilities:

• Upload videos up to 10 GB directly to Azure Blob Storage
• Describe what to extract in natural language via a chat interface
• AI agents (planner → analysis → processing) orchestrate the full pipeline
• FFmpeg keyframe pre-processing dramatically reduces AI token costs
• Output videos delivered via signed CDN URLs and email notification
• All processing services auto-scale to zero when idle (KEDA on Azure Container Apps)

How It Works

Upload video (Angular → Blob Storage via SAS token)
    ↓
Pre-processing worker (FFmpeg keyframe extraction → PostgreSQL index)
    ↓
User submits prompt (LibreChat iframe → API Gateway → Agent Orchestrator)
    ↓
CrewAI crew: Planner → Analysis Agent (MCP tools) → Processing Agent (MCP tools)
    ↓
Output video written to Blob Storage
    ↓
Signed download URL delivered via SSE stream + email notification

All steps are asynchronous and fault-tolerant via Azure Service Bus queues.

External Agents

The platform's MCP tools can be used directly from Claude Desktop or the LibreChat official image via an MCP bridge (port 8300) that translates standard MCP JSON-RPC to the platform's SSE tool protocol.

Claude Desktop:

# Start MCP bridge
bash external-agents/claude-desktop/scripts/start-mcp-bridge.sh

# Install config (Windows PowerShell)
.\external-agents\claude-desktop\scripts\install.ps1
# Restart Claude Desktop — Tools icon should show 19 tools

| Session started — upload link provided | Job complete — extraction summary and download link | |---|---| | | |

LibreChat (official image):

cp external-agents/librechat/.env.example external-agents/librechat/.env
# Set ANTHROPIC_API_KEY and generate random secrets (see docs/getting-started.md §13.2)
cd external-agents/librechat && docker compose up -d
# Open http://localhost:3081

| Agent running MCP tool calls (ingest → detect → clip) | Extraction complete — final output URL | |---|---| | | |

See docs/getting-started.md § External agents for the full walkthrough.

Tech Stack

| Layer | Technology | |---|---| | Frontend | Angular 19 + LibreChat (forked, iframe embed) | | API / BFF | Node.js + Express (TypeScript) | | AI Orchestration | Python + CrewAI + FastAPI | | LLM | Any LiteLLM-compatible model (Anthropic Claude, OpenAI, AWS Bedrock, and more) | | Tool Protocol | MCP over SSE transport | | Container Platform | Azure Container Apps + KEDA | | Infrastructure as Code | Terraform | | Storage | Azure Blob Storage | | Database | PostgreSQL 15 (ACA container, Azure Files backed) | | Messaging | Azure Service Bus | | Auth | Azure Entra External ID (magic link / JWT) | | Local Dev Emulation | Docker Compose + Azurite | | CI/CD | GitLab CI (mirrored to GitHub) |

Quick Start (Local Dev)

Requires Docker Desktop (≥ 4.30) with WSL 2. See docs/getting-started.md for full prerequisites and Azure setup.

1. Copy environment files:

cp backend/api-gateway/.env.example              backend/api-gateway/.env
cp backend/agent-orchestrator/.env.example       backend/agent-orchestrator/.env
cp backend/preprocessing-worker/.env.example     backend/preprocessing-worker/.env
cp mcp-servers/mcp-server-analysis/.env.example  mcp-servers/mcp-server-analysis/.env
cp mcp-servers/mcp-server-processing/.env.example mcp-servers/mcp-server-processing/.env
cp frontend/librechat/.env.example               frontend/librechat/.env

Edit backend/agent-orchestrator/.env and set ANTHROPIC_API_KEY.

2. Start the stack:

cd infrastructure/docker-compose
docker compose up --build

3. Create Service Bus queues (once, after stack is up):

export SERVICE_BUS_CONNECTION_STRING="Endpoint=sb://localhost;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;"
python scripts/create_service_bus_queues.py

4. Verify services:

curl http://localhost:8000/health   # API Gateway
curl http://localhost:8001/health   # Agent Orchestrator
curl http://localhost:8100/tools    # MCP Analysis tools
curl http://localhost:8200/tools    # MCP Processing tools

Services run on:

| Service | Port | |---|---| | Angular Shell | http://localhost:4200 | | LibreChat | http://localhost:3080 | | API Gateway | http://localhost:8000 | | Agent Orchestrator | http://localhost:8001 | | MCP Analysis | http://localhost:8100 | | MCP Processing | http://localhost:8200 | | Azurite (Blob) | http://localhost:10000 | | PostgreSQL | localhost:5433 |

Running Tests

E2E tests (fully containerised)

scripts/run-e2e-local.sh
# With frontier vision tools:
ANTHROPIC_API_KEY=sk-... scripts/run-e2e-local.sh

Documentation

| Document | Description | |---|---| | docs/architecture.md | System design, data flows, service responsibilities, component details, deployment diagrams | | docs/getting-started.md | Full setup guide — prerequisites, GitLab/GitHub/Azure configuration, local dev bootstrap, CI/CD variables, secrets reference, troubleshooting | | docs/local-development.md | Day-to-day local development — starting the stack, running services and tests, common tasks | | docs/e2e-tests.md | End-to-end pipeline tests | | docs/azure-production-deployment.md | Azure production deployment reference — services, roles, inter-service communication, scaling, CI/CD | | docs/azure-credentials.md | Azure credentials setup — every credential the platform needs, how to create and configure each | | docs/terraform.md | Terraform layout, modules, environments, and how the pieces connect | | docs/ai-containers-deep-dive.md | Deep dive into each AI container — inputs, outputs, and position in the job processing sequence | | docs/gitlab-pipeline.md | CI/CD pipeline — every stage and job, environment lifecycle, and SDLC workflow | | docs/instant-compilation-errors.md | Getting immediate type and syntax error feedback during local development without Docker rebuilds | | docs/local-containers-report.md | Local container architecture report | | external-agents/claude-desktop/README.md | Claude Desktop MCP integration | | external-agents/librechat/README.md | LibreChat official image MCP integration |

Repository Structure

backend/
  api-gateway/            Node.js + Express (TypeScript) — auth, SAS tokens, SSE, chat proxy
  agent-orchestrator/     Python + CrewAI (FastAPI) — planner/analyst/processor agents
  preprocessing-worker/   Python — FFmpeg keyframe extraction

mcp-servers/
  mcp-server-analysis/    Port 8100 — ingest_video, extract_frames, detect_motion, detect_motion_sports,
                          detect_objects, detect_objects_vision, analyze_scene, transcribe_audio,
                          estimate_height_above_surface, read_asset, query_asset, write_query_asset,
                          write_segments_asset
  mcp-server-processing/  Port 8200 — split_video, extract_clip, extract_clips_bulk, merge_clips,
                          transform_video, write_asset, query_asset, write_query_asset

frontend/
  angular-shell/          Angular 19 — upload UI, job dashboard, LibreChat iframe host
  librechat/              Forked LibreChat — custom endpoint, branding, job status postMessage bridge

external-agents/
  mcp-bridge/             Standard MCP server (port 8300) — SSE + stdio transports
  claude-desktop/         Claude Desktop config + install scripts
  librechat/              LibreChat official image stack
  agent-instructions/     System prompt for external agents

infrastructure/
  docker-compose/         Full local dev stack
  terraform/
    modules/              aca, storage, database (reusable modules)
    envs/                 dev, test (ephemeral per CI pipeline)

tests/
  e2e/                    End-to-end tests (ephemeral Azure + local Docker Compose)

scripts/
  init_db.py                    Create all database tables
  init_storage.py               Create Blob Storage containers
  create_service_bus_queues.py  Create all Service Bus queues
  run-e2e-local.sh              Run E2E tests locally (fully containerised)
  bootstrap-dev.sh              First-time local dev setup
  smoke-test.sh                 Quick smoke test against running stack
  teardown.sh                   Stop and clean up local stack
  repair_job_output.py          Repair job output records in PostgreSQL
  collect_test_logs.py          Collect logs from CI test run
  check_e2e_threshold.py        Assert E2E test pass rate meets threshold

docs/
  architecture.md         System architecture reference
  getting-started.md      Full setup and deployment guide