🔍 CompeteFlow — Local Competitive Intelligence Agent

A fully local, multi-agent AI system built with CrewAI + Ollama that autonomously gathers, analyses, and reports on competitor intelligence — no cloud APIs, no SaaS subscriptions, no data leaving your machine.

---

🎯 What CompeteFlow Does

CompeteFlow runs a CrewAI multi-agent pipeline that autonomously:

| Capability | What it does | Tools used | |---|---|---| | 🌐 Monitor | Scrapes competitor websites, extracts pricing tiers, detects changes from previous snapshots | httpx, BeautifulSoup4, SQLite | | 📰 Track | Fetches news articles, RSS posts, and Reddit discussions — classifies sentiment | NewsAPI, feedparser, PRAW | | 🔍 Analyse | RAG-powered feature extraction, competitive comparison, gap analysis | ChromaDB, sentence-transformers, Ollama | | 📊 Report | Generates battle cards, executive summaries, and feature matrices in markdown | Ollama, pathlib |

Everything runs locally on your Windows laptop. No OpenAI API calls. No cloud database. One streamlit run app.py and you have a full competitive intelligence dashboard.

---

🏗️ Architecture

CrewAI Multi-Agent Pipeline

╔══════════════════════════════════════════════════════════════════╗
║                     CompeteFlow Crew                             ║
║                  (CrewAI Sequential Process)                     ║
╠══════════════════════════════════════════════════════════════════╣
║                                                                  ║
║  ┌─────────────────────────────────────────────────────────┐    ║
║  │  AGENT 1 — WebScraperAgent                              │    ║
║  │  Role: Competitor Website Intelligence Specialist        │    ║
║  │  Tools: scrape_website, extract_pricing,                │    ║
║  │         detect_changes, store_snapshot                  │    ║
║  └───────────────────────┬─────────────────────────────────┘    ║
║                          │ output (website intel)               ║
║                          ▼                                       ║
║  ┌─────────────────────────────────────────────────────────┐    ║
║  │  AGENT 2 — NewsTrackerAgent                             │    ║
║  │  Role: Competitive News and Social Intelligence Analyst  │    ║
║  │  Tools: fetch_news_api, fetch_rss_feeds,                │    ║
║  │         fetch_reddit_mentions, classify_sentiment        │    ║
║  └───────────────────────┬─────────────────────────────────┘    ║
║                          │ output (news + sentiment)            ║
║                          ▼ (+ website context)                  ║
║  ┌─────────────────────────────────────────────────────────┐    ║
║  │  AGENT 3 — AnalyzerAgent                                │    ║
║  │  Role: Competitive Intelligence Analyst                  │    ║
║  │  Tools: embed_and_store, retrieve_context,              │    ║
║  │         extract_features, compare_competitors,           │    ║
║  │         identify_gaps                                   │    ║
║  └───────────────────────┬─────────────────────────────────┘    ║
║                          │ output (full analysis)               ║
║                          ▼                                       ║
║  ┌─────────────────────────────────────────────────────────┐    ║
║  │  AGENT 4 — ReportAgent                                  │    ║
║  │  Role: Competitive Intelligence Report Writer            │    ║
║  │  Tools: generate_battle_card, generate_executive_summary│    ║
║  │         generate_feature_matrix, export_report          │    ║
║  └───────────────────────┬─────────────────────────────────┘    ║
║                          │                                       ║
║                          ▼                                       ║
║              ⚔️  Battle Card + Executive Summary                 ║
╚══════════════════════════════════════════════════════════════════╝

APScheduler Automation

┌─────────────────────────────────────────────────┐
│  BackgroundScheduler (daemon thread)             │
│                                                  │
│  Weekly (Mon 8am UTC)                            │
│    └─► Full 4-agent pipeline, all competitors   │
│                                                  │
│  Daily (9:30am UTC)                              │
│    └─► News-only run, all competitors            │
│                                                  │
│  Manual (Streamlit "Run Now" button)             │
│    └─► Full or news run, single competitor      │
└─────────────────────────────────────────────────┘

---

🛠️ Tech Stack

| Layer | Tool | Version | Why | |---|---|---|---| | Agent Framework | CrewAI | 0.80.0 | Purpose-built multi-agent orchestration, @tool decorator, context passing | | LLM | Ollama (mistral / phi3 / llama3) | latest | Fully local, free, CrewAI-compatible via LiteLLM | | LLM Bridge | LiteLLM | 1.51.3 | CrewAI's internal bridge — routes ollama/mistral to localhost | | Web Scraping | httpx + BeautifulSoup4 | 0.27 / 4.12 | Sync Client() — Windows-safe inside @tool functions | | News | newsapi-python + feedparser | 0.2.7 / 6.0 | NewsAPI (100 req/day free) + RSS (unlimited) | | Social | PRAW | 7.7.1 | Reddit read-only, free with Reddit account | | Vector Store | ChromaDB | 0.5.18 | Local persistent vector DB, pathlib-safe on Windows | | Embeddings | sentence-transformers | 3.3.1 | Local all-MiniLM-L6-v2, no API needed | | Structured DB | SQLite (stdlib) | — | Zero config, single file, 10-table schema | | Scheduler | APScheduler | 3.10.4 | BackgroundScheduler — Windows-safe daemon thread | | Dashboard | Streamlit | 1.39.0 | Multi-page app with Plotly charts and RAG chat | | Charts | Plotly | 5.24.1 | Price history and mention volume charts | | Config | python-dotenv | 1.0.1 | .env → os.environ, typed config.py |

---

💻 System Requirements

| Requirement | Minimum | Recommended | |---|---|---| | OS | Windows 10 64-bit | Windows 11 64-bit | | Python | 3.10 | 3.11 | | RAM | 8 GB | 16 GB | | Disk | 10 GB free | 20 GB free | | CPU | 4 cores | 8 cores (Ollama runs faster) | | GPU | Not required | NVIDIA (Ollama uses it automatically if present) | | Network | Required for NewsAPI / Reddit / scraping | — |

Mac / Linux: CompeteFlow works on macOS and Linux with minor adjustments:

• Replace venv\Scripts\activate with source venv/bin/activate
• Ollama runs as ollama serve in terminal (not a background service)
• All pathlib.Path() usage is cross-platform — no changes needed

---

⚡ Quick Start (Windows)

Step 1 — Install Ollama

1. Download from ollama.ai/download/windows
2. Run the installer — Ollama starts automatically in the system tray
3. Open a terminal and pull the recommended model:

ollama pull mistral

Why mistral? Best balance of reasoning quality and speed on CPU-only hardware (7B parameters, ~4GB RAM). Alternatives: phi3 (3.8B, fastest), llama3 (8B, highest quality, slower).

---

Step 2 — Clone and Set Up Python Environment

git clone https://github.com/YOUR_USERNAME/competeflow.git
cd competeflow
python -m venv venv
venv\Scripts\activate

PowerShell (if activate is blocked):

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
venv\Scripts\Activate.ps1

---

Step 3 — Install Dependencies

pip install --upgrade pip

REM Install PyTorch CPU-only first (saves ~2GB vs GPU version)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

REM Install all other dependencies
pip install -r requirements.txt

Note: First install takes 5-10 minutes. sentence-transformers downloads ~90MB model on first run.

---

Step 4 — Configure Environment

copy .env.example .env

Open .env in any editor and fill in:

# Required — CrewAI needs this even for local Ollama (set to NA)
OPENAI_API_KEY=NA

# Ollama (already running from Step 1)
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_MODEL=mistral

# Optional — NewsAPI (free tier, 100 req/day)
# Sign up at: https://newsapi.org/register
NEWSAPI_KEY=your_key_here

# Optional — Reddit API (free read-only)
# Create app at: https://www.reddit.com/prefs/apps (type: script)
REDDIT_CLIENT_ID=your_client_id
REDDIT_CLIENT_SECRET=your_client_secret
REDDIT_USER_AGENT=windows:CompeteFlow/1.0 (by u/your_username)

CompeteFlow works without NewsAPI and Reddit — RSS feeds are always available as fallback.

---

Step 5 — Verify Setup

python test_setup.py

Expected output:

✅  Python 3.11.x (3.10+ required)
✅  crewai 0.80.0 installed and importable
✅  Ollama running at http://localhost:11434
✅  ChromaDB: write and query successful
✅  all-MiniLM-L6-v2 loaded in 2.3s
⚠️  NewsAPI: key not configured (optional)
⚠️  Reddit PRAW: credentials not configured (optional)
✅  SQLite working — pathlib.Path() pattern verified
✅  config.py loaded successfully

7 green ✅ = ready. The 2 optional ⚠️ items (NewsAPI, Reddit) can be added anytime.

---

Step 6 — Run the Dashboard

streamlit run app.py

Open http://localhost:8501 in your browser.

On first launch:

• Notion, Airtable, and Coda are pre-loaded as demo competitors
• Click 🚀 Run Now on any competitor to start the first crew pipeline
• Watch the terminal for the CrewAI agent reasoning output (verbose mode)
• Battle card appears in the Reports tab when complete (~5-15 min on CPU)

---

Step 7 — (Optional) Test Individual Components

REM Test tools in isolation (no Crew, no Ollama LLM needed)
python tools\test_tools.py --no-ollama --no-network

REM Test with Ollama (classify_sentiment, extract_features)
python tools\test_tools.py --chroma

REM Test crew assembly without running (no Ollama needed)
python crew\competeflow_crew.py --build-only

REM Run full pipeline test with Notion as demo
python test_crew.py

---

📁 Project Structure

competeflow/
│
├── app.py                     # Streamlit dashboard (5 pages)
├── scheduler.py               # APScheduler automation engine
├── config.py                  # Central config — all settings via pathlib.Path
├── requirements.txt           # Pinned dependencies (Windows-tested)
├── .env.example               # Environment variable template
├── test_setup.py              # 9-check environment verifier
├── test_crew.py               # End-to-end crew pipeline test
│
├── agents/
│   └── crew_agents.py         # 4 CrewAI Agent definitions
│
├── tasks/
│   └── crew_tasks.py          # 4 CrewAI Task definitions + context wiring
│
├── crew/
│   └── competeflow_crew.py    # Crew assembly, kickoff, batch runner
│
├── tools/
│   ├── scraper_tools.py       # @tool: scrape_website, extract_pricing, ...
│   ├── news_tools.py          # @tool: fetch_news_api, classify_sentiment, ...
│   ├── chroma_tools.py        # @tool: embed_and_store, retrieve_context, ...
│   ├── report_tools.py        # @tool: generate_battle_card, export_report, ...
│   └── test_tools.py          # Standalone tool test runner
│
├── storage/
│   ├── database.py            # SQLite layer — 7 repos, 10-table schema
│   └── chroma_store.py        # ChromaDB manager — semantic search, RAG context
│
├── data/                      # Created at runtime — gitignored
│   ├── competeflow.db         # SQLite database
│   ├── chroma_db/             # ChromaDB vector store
│   └── snapshots/             # Raw website snapshots
│
└── reports/                   # Generated battle cards — gitignored
    ├── notion/
    │   ├── notion_battle_card.md
    │   └── notion_exec_summary.md
    └── airtable/
        └── airtable_battle_card.md

---

📊 Example Output — Notion Battle Card

# Notion — Battle Card
*Generated: 2025-01-20 | Source: CompeteFlow Intelligence*

## Who They Are
Notion is an all-in-one connected workspace that replaces wikis, docs,
project management, and databases with a single block-based editor.
Serving 30M+ users across SMB and mid-market segments.

## Their Positioning
"The connected workspace where better, faster work happens."
Targets knowledge workers who want to consolidate fragmented tools.

## Pricing
| Tier       | Price           | Key Feature Limit        |
|------------|-----------------|--------------------------|
| Free       | $0              | Up to 10 guests          |
| Plus       | $10/user/month  | Unlimited blocks + guests|
| Business   | $15/user/month  | Advanced analytics, SAML |
| Enterprise | Custom          | Dedicated CSM, SLA       |
| Notion AI  | +$10/user/month | AI writing add-on        |

## Top Features
- Block-based editor (text, databases, kanban, calendar, gallery)
- Relational databases with formula and rollup support
- Collaborative wikis with permission inheritance
- Notion AI (writing assistant, summarisation, Q&A)
- 100+ native integrations + API
- Template gallery (10,000+ community templates)

## Their Strengths
- Extremely flexible — adapts to almost any workflow
- Massive template library lowers onboarding friction
- Strong brand and community (30M users, active Reddit)
- AI features integrated into existing pricing (not a separate product)

## Their Weaknesses
- Performance degrades with large databases (10k+ rows)
- No native time tracking or resource management
- API rate limits (3 req/sec) block enterprise automation use cases
- Complex for non-technical users — high setup overhead

## Recent Strategic Moves
- Launched Notion AI Q&A (ask questions across entire workspace)
- Expanded Enterprise tier with HIPAA compliance (targeting healthcare)
- Acquired Skiff (privacy-focused workspace) — signals E2E encryption roadmap

## How To Win Against Them
1. **Lead with performance:** "Unlike Notion, our database handles 500k+ rows
   without slowing down — benchmark test available"
2. **Lead with time tracking:** "Notion has no native time tracking — your team
   needs a third-party integration and another monthly bill"
3. **Lead with simplicity for ops teams:** "Notion's flexibility is also its
   weakness — our opinionated templates get ops teams running in a day, not a month"

## Objection Handling
| Prospect says | You say |
|---|---|
| "We already use Notion" | "What's your biggest pain point with it? Performance? We benchmark 10x faster on large databases." |
| "Notion is cheaper" | "Notion Plus is $10/user + $10 AI add-on = $20. Our Business plan is $18 all-in with AI included." |
| "Our team loves Notion's flexibility" | "Flexibility without guardrails creates chaos at scale. Our structured workflows prevent that." |

## Key Quotes to Use with Prospects
- "Notion is a blank canvas — great for one person, chaotic for a 50-person team."
- "Notion AI is an add-on. Our AI is built in at no extra cost."
- "We've seen 3 Notion migrations this quarter — all citing database performance."

---

🔧 Configuration

All settings live in config.py and are loaded from .env. Key settings:

| Variable | Default | Description | |---|---|---| | OLLAMA_MODEL | mistral | LLM model for all agents | | OLLAMA_BASE_URL | http://localhost:11434 | Ollama server address | | EMBEDDING_MODEL | all-MiniLM-L6-v2 | sentence-transformers model | | CHUNK_SIZE | 500 | Characters per ChromaDB chunk | | TOP_K_RESULTS | 5 | RAG retrieval results | | SCHEDULE_FULL_RUN_DAY | monday | Day for weekly full run | | SCHEDULE_FULL_RUN_HOUR | 8 | Hour (UTC) for weekly run | | SCHEDULE_NEWS_HOUR | 9 | Hour (UTC) for daily news run | | CREW_VERBOSE | True | Show agent reasoning in terminal | | DEMO_COMPETITORS | Notion,Airtable,Coda | Pre-seeded competitors |

---

🐛 Troubleshooting

OPENAI_API_KEY error on startup

AuthenticationError: No auth credentials found

Fix: Add OPENAI_API_KEY=NA to your .env file. CrewAI requires this variable to be set even when using local Ollama — it checks for presence, not validity.

---

Ollama timeout in CrewAI tools

httpx.ReadTimeout: timed out

Fix: Increase timeout in .env:

SCRAPE_TIMEOUT_SECONDS=60

Or set request_timeout=120.0 in the Ollama() constructor in config.py.

---

ChromaDB database is locked error

sqlite3.OperationalError: database is locked

Fix: This occurs when multiple processes hold ChromaDB connections simultaneously. CompeteFlow creates fresh clients per tool call to prevent this, but if you see it during development, close other Python processes accessing data/chroma_db/.

---

PowerShell activation blocked

venv\Scripts\Activate.ps1 cannot be loaded because running scripts is disabled

Fix:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

---

crewai.LLM not found (older CrewAI version)

CompeteFlow's get_llm() in config.py auto-detects your CrewAI version and uses the correct LLM wrapper:

• CrewAI ≥ 0.70: uses crewai.LLM(model="ollama/mistral")
• CrewAI < 0.70: uses langchain_community.llms.Ollama(model="mistral")

No manual fix needed — check your version with pip show crewai.

---

First run is very slow

Expected. On first run, sentence-transformers downloads all-MiniLM-L6-v2 (~90MB) to C:\Users\<you>\.cache\huggingface\. Subsequent runs load from cache in ~2 seconds. Ollama also pulls the model on first use if not already pulled.

---

🗺️ Roadmap

• [ ] storage/database.py — migration runner for schema v2+
• [ ] Hierarchical process mode with manager agent (Phase 2 upgrade)
• [ ] Export battle cards to HTML + PDF
• [ ] Email digest integration (weekly report via SMTP)
• [ ] Docker compose.yml for non-Windows deployment
• [ ] GitHub Actions workflow for automated test runs
• [ ] test_crew.py with mock HTTP responses (offline CI)
• [ ] Streamlit sharing deployment guide

---

🤝 Contributing

See CONTRIBUTING.md for development setup, coding standards, and how to add new agents, tools, or data sources.

---

📄 License

MIT License — see LICENSE.

---

🙏 Acknowledgements

• CrewAI — the multi-agent framework that makes this possible without hundreds of lines of orchestration boilerplate
• Ollama — local LLM inference that makes the whole stack free
• ChromaDB — the easiest local vector store for RAG
• sentence-transformers — local embeddings, no API needed
• Streamlit — turns Python into a real dashboard in hours

---

Built as a portfolio project to demonstrate real-world agentic AI engineering with CrewAI, Ollama, ChromaDB, and Streamlit on Windows.