CrewAI Data Analyst - Full-Stack AI Platform

A production-ready, full-stack data analysis platform built with CrewAI, featuring specialized AI agents that collaborate to perform end-to-end data analysis tasks. Complete with a modern web interface, multiple LLM provider support, and advanced workflow management.

🎯 Overview

This project creates a comprehensive AI-powered data analysis platform with both a powerful backend API and an intuitive web interface. The system features specialized AI agents working together to analyze data systematically and professionally, with support for multiple LLM providers including Gemini, Groq, OpenAI, and Anthropic.

✨ Full-Stack Features

• 🌐 Modern Web Interface: Clean, intuitive React-based frontend
• ⚡ FastAPI Backend: High-performance API with async support
• 🤖 Multiple LLM Support: Gemini, Groq, OpenAI, Anthropic
• 🌊 Flow-Based Workflows: Structured execution with state management
• 📊 Real-Time Monitoring: Live progress tracking and job management
• � Firecrawl Integration: Advanced web scraping and search capabilities
• 📁 File Management: Drag-and-drop uploads with format validation
• � Interactive Dashboards: Visual progress and results display
• � Production Ready: Comprehensive error handling and logging

👥 Team Members

The Data Analyst Crew consists of 6 specialized agents:

1. Data Collection Specialist 🗂️

   • Gathers and validates data from various sources
   • Performs data quality assessment and cleaning
   • Prepares datasets for analysis

2. Statistical Analyst 📊

   • Conducts advanced statistical analysis
   • Performs hypothesis testing and regression analysis
   • Identifies patterns and correlations

3. Data Visualization Expert 📈

   • Creates compelling visualizations and charts
   • Builds interactive dashboards
   • Transforms data into visual insights

4. Business Intelligence Analyst 💼

   • Translates data insights into business recommendations
   • Provides strategic analysis and actionable insights
   • Researches industry benchmarks and context

5. Quality Assurance Analyst ✅

   • Validates analysis methods and results
   • Ensures data integrity and accuracy
   • Identifies limitations and potential biases

6. Technical Report Writer 📝

   • Synthesizes all findings into comprehensive reports
   • Creates executive summaries for stakeholders
   • Ensures clear communication of results

🛠️ Quick Setup

Option 1: Automated Full-Stack Setup (Recommended)

python scripts/setup.py

This automated script will:

• Check system requirements (Python 3.9+, Node.js)
• Set up Python virtual environment
• Install all backend dependencies
• Install frontend dependencies and build the React app
• Create necessary directories and configuration files
• Generate run scripts for easy startup

Option 2: Manual Setup

See our detailed Installation Guide for step-by-step instructions.

3. Add API Keys

Choose your preferred LLM provider and add the corresponding API key to your .env file:

# For Gemini (recommended for cost-effective analysis)
GEMINI_API_KEY=your_gemini_api_key_here
MODEL=gemini-2.0-flash-exp

# For Groq (high-performance option)
# GROQ_API_KEY=your_groq_api_key_here
# MODEL=llama-3.2-90b-text-preview

# Optional: Web search and scraping capability (Firecrawl)
FIRECRAWL_API_KEY=your_firecrawl_api_key_here

🚀 Running the Application

Quick Start

# Start the full application
python main.py

Development Mode

# Start backend and frontend separately
# Backend:
python main.py

# Frontend (in another terminal):
cd frontend && npm run dev

Access Points

• Web Interface: http://localhost:3000 (development) or http://localhost:8000 (production)
• API Backend: http://localhost:8000
• API Documentation: http://localhost:8000/api/docs
• Alternative Docs: http://localhost:8000/api/redoc

🔥 Firecrawl Integration

This project uses Firecrawl for advanced web scraping and search capabilities, replacing traditional search APIs with a more robust solution.

Why Firecrawl?

• Clean Output: Converts websites to clean markdown format
• JavaScript Support: Handles dynamic content and SPAs
• Rate Limiting: Built-in intelligent rate limiting
• Structured Data: Returns properly formatted, structured data
• Cost Effective: More predictable pricing than search alternatives

Firecrawl Setup

1. Sign up at firecrawl.dev
2. Get your API key from the dashboard
3. Add it to your .env file:

FIRECRAWL_API_KEY=your_actual_firecrawl_api_key_here

Firecrawl Features Used

• FirecrawlSearchTool: Web search across multiple sources
• FirecrawlScrapeWebsiteTool: Deep content extraction from websites
• Automatic Content Cleaning: Removes ads, navigation, and clutter
• Markdown Conversion: Clean, readable format for AI processing

🚀 Usage Examples

Standard Analysis with Gemini

from data_analyst_crew import create_production_crew, ModelProvider, AnalysisType

# Create crew with Gemini
crew = create_production_crew(
    provider=ModelProvider.GEMINI,
    model="gemini-2.0-flash-exp",
    enable_async=True
)

# Run analysis
result = crew.run_analysis(
    data_source="your_data.csv",
    analysis_type=AnalysisType.COMPREHENSIVE
)

High-Performance Analysis with Groq

# Create crew with Groq for faster processing
crew = create_production_crew(
    provider=ModelProvider.GROQ,
    model="llama-3.2-90b-text-preview",
    enable_async=True
)

result = crew.run_analysis(
    data_source="your_data.csv",
    analysis_type=AnalysisType.COMPREHENSIVE
)

Flow-Based Analysis with Quality Gates

from data_analysis_flow import create_analysis_flow

# Create flow-based workflow
flow = create_analysis_flow(
    provider=ModelProvider.GEMINI,
    model="gemini-2.0-flash-exp"
)

# Execute with quality gates and state management
final_state = flow.kickoff(
    data_source="your_data.csv",
    analysis_type=AnalysisType.COMPREHENSIVE
)

Asynchronous Analysis

import asyncio

async def run_async_analysis():
    crew = create_production_crew(
        provider=ModelProvider.GEMINI,
        enable_async=True
    )
    
    result = await crew.run_analysis_async(
        data_source="your_data.csv",
        analysis_type=AnalysisType.COMPREHENSIVE
    )
    return result

# Run async analysis
result = asyncio.run(run_async_analysis())

📊 Analysis Types

• COMPREHENSIVE: Full analysis with all agents
• QUICK: Streamlined analysis for faster results
• TIME_SERIES: Specialized temporal data analysis
• PREDICTIVE: Focus on forecasting and predictions
• EXPLORATORY: Open-ended data exploration

🏗️ Project Structure

crewAI-data-analyst/
├── main.py                        # Application launcher
├── requirements.txt               # Python dependencies
├── .env.production               # Environment template
├── README.md                     # This file
├── src/                          # Source code
│   ├── __init__.py              # Package initialization
│   ├── api/                     # FastAPI backend
│   │   ├── __init__.py
│   │   └── main.py              # API routes and server
│   ├── crew/                    # AI crew implementations
│   │   ├── __init__.py
│   │   ├── data_analyst_crew.py # Main crew logic
│   │   └── data_analysis_flow.py # Flow-based workflows
│   └── utils/                   # Utility functions
│       └── __init__.py          # Helper functions and config
├── scripts/                     # Setup and utility scripts
│   ├── setup.py                # Automated setup
│   └── example_analysis.py     # Usage examples
├── frontend/                    # React web interface
│   ├── src/
│   │   ├── components/          # UI components
│   │   ├── pages/              # Application pages
│   │   └── services/           # API integration
│   ├── package.json
│   └── tailwind.config.js
├── config/                      # Configuration files
│   ├── agents.yaml             # Agent definitions
│   └── tasks.yaml              # Task configurations
├── docs/                       # Documentation
│   └── installation.md        # Setup guide
├── tests/                      # Test files
│   ├── __init__.py
│   └── test_basic.py          # Basic unit tests
├── uploads/                    # User uploaded files
├── outputs/                    # Analysis results
└── static/                     # Built frontend assets

⚙️ Configuration

LLM Provider Configuration

The system supports multiple providers with specific optimizations:

| Provider | Best For | Model Recommendation | |----------|----------|----------------------| | Gemini | Cost-effective, comprehensive analysis | gemini-2.0-flash-exp | | Groq | High-performance, fast processing | llama-3.2-90b-text-preview | | OpenAI | Reliable, well-tested workflows | gpt-4o | | Anthropic | Complex reasoning, safety-focused | claude-3-sonnet-20240229 |

Production Settings

# Advanced configuration
crew_config = CrewConfig(
    llm_config=LLMConfig(
        provider=ModelProvider.GEMINI,
        model="gemini-2.0-flash-exp",
        temperature=0.7,
        max_tokens=4000,
        timeout=120
    ),
    enable_memory=True,
    enable_async=True,
    max_retries=3,
    output_dir="outputs",
    enable_monitoring=True
)

📈 Output Examples

The crew generates comprehensive outputs:

1. Quality Assessment Report - Data validation and cleaning summary
2. Statistical Analysis Report - Detailed statistical findings
3. Interactive Visualizations - Charts, graphs, and dashboards
4. Business Intelligence Report - Strategic insights and recommendations
5. Quality Assurance Report - Validation and reliability assessment
6. Executive Summary - High-level findings for stakeholders
7. Performance Metrics - Execution time, success rates, error tracking

🔧 Production Features

Error Handling & Monitoring

• Comprehensive logging with structured output
• Automatic retry mechanisms for failed tasks
• Performance metrics tracking
• Error categorization and reporting

Quality Gates & Validation

• Data quality scoring (0-100)
• Workflow decision points based on quality
• Statistical validation of results
• Bias detection and reporting

Scalability & Performance

• Asynchronous task execution
• Memory management for large datasets
• Resource usage optimization
• Parallel processing capabilities

🚀 Advanced Usage

Custom Workflows

# Create custom analysis workflow
def custom_analysis_workflow(data_source: str):
    # Step 1: Quick data assessment
    crew = create_production_crew(
        provider=ModelProvider.GEMINI,
        enable_async=True
    )
    
    quick_result = crew.run_analysis(
        data_source=data_source,
        analysis_type=AnalysisType.QUICK
    )
    
    # Step 2: Comprehensive analysis if quality is good
    if quick_result and quick_result.get('data_quality_score', 0) > 70:
        comprehensive_result = crew.run_analysis(
            data_source=data_source,
            analysis_type=AnalysisType.COMPREHENSIVE
        )
        return comprehensive_result
    
    return quick_result

Integration with External Systems

# Database integration
def analyze_database_table():
    import pandas as pd
    from sqlalchemy import create_engine
    
    # Load data from database
    engine = create_engine('your_database_url')
    df = pd.read_sql('SELECT * FROM your_table', engine)
    df.to_csv('temp_data.csv', index=False)
    
    # Analyze with crew
    crew = create_production_crew(provider=ModelProvider.GEMINI)
    return crew.run_analysis('temp_data.csv')

🔍 Troubleshooting

Common Issues

1. API Key Errors

   # Verify your API keys
   python -c "from dotenv import load_dotenv; load_dotenv(); import os; print('GEMINI_API_KEY' in os.environ)"
   

2. Memory Issues with Large Datasets

   # Use data sampling for initial analysis
   df_sample = df.sample(n=10000)  # Sample 10k rows
   

3. Timeout Issues

   # Increase timeout in configuration
   llm_config = LLMConfig(timeout=300)  # 5 minutes
   

� API Reference

Core Classes

• ProductionDataAnalystCrew: Main crew orchestrator
• DataAnalysisFlow: Flow-based workflow system
• ModelProvider: Enum for LLM providers
• AnalysisType: Enum for analysis types
• LLMConfig: LLM configuration dataclass
• CrewConfig: Crew configuration dataclass

Factory Functions

• create_production_crew(): Create configured crew
• create_analysis_flow(): Create flow-based workflow

📄 License

This project is open source and available under the MIT License.

🤝 Contributing

Feel free to contribute by:

• Adding new LLM providers
• Improving analysis capabilities
• Enhancing visualizations
• Adding new workflow types
• Improving error handling

---

🚀 Ready to Get Started

Your CrewAI data analyst team is ready for production use with advanced AI-powered analysis capabilities and Firecrawl web scraping integration! 📊🤖