HR Onboarding Agent

🚀 Fully automated HR onboarding with CrewAI agents + vision-based browser automation (no DOM manipulation)

An AI-powered HR onboarding automation system that uses CrewAI multi-agent orchestration and Computer Use Agent (CUA) with vision-based browser automation to streamline the employee onboarding process.

Overview

This project demonstrates a fully automated HR onboarding workflow where AI agents:

1. Parse employee data from CSV uploads
2. Automatically assign seats and equipment by controlling a web-based HR system through visual recognition
3. Generate and send notification emails to relevant stakeholders

The system uses the MS-FARA 7B vision model running locally via LM Studio to analyze screenshots and determine the next UI actions, enabling true computer use capabilities without relying on DOM manipulation or API calls.

Architecture

The system consists of 5 main layers:

1. User Interface Layer: HR System (upload) + Onboarding System (target UI)
2. CrewAI Orchestration Layer: 3 specialized agents working in sequence
3. Tools Layer: CUA Tool + Email Sender Tool
4. CUA Engine Layer: Playwright + MS-FARA 7B vision model
5. Data Layer: JSON storage for employee data, seats, assets, and emails

For detailed architecture documentation, see docs/ARCHITECTURE.md.

✨ Features

• 🤖 Multi-Agent Orchestration: Three specialized agents working in sequence using CrewAI
• 👁️ Vision-Based Browser Automation: No DOM manipulation - the AI "sees" the screen and decides what to click
• 📊 Real-Time Execution Logs: Watch the AI's progress in real-time through the web interface
• ⏸️ Cancellable Operations: Stop the onboarding process at any time
• 📧 Email Notifications: Automatic email generation for HR, managers, and IT teams
• 🎯 CUA-Friendly UI: Large click targets and clear visual indicators optimized for AI vision

Tech Stack

| Component | Technology | |-----------|------------| | Multi-Agent Framework | CrewAI 1.7.0 | | Vision Model | MS-FARA 7B | | Local LLM Server | LM Studio | | Browser Automation | Playwright (Python) | | Web Framework | Flask | | Frontend | Vanilla HTML/CSS/JS |

Prerequisites

• Python 3.11+
• LM Studio with MS-FARA 7B model loaded
• Playwright browser automation library

📦 Installation

1. Clone the Repository

git clone https://github.com/gspain89/crewai-hr-onboarding-cua-agent.git
cd crewai-hr-onboarding-cua-agent

2. Set Up CrewAI Environment

cd hr_onboarding_crew
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install -e .
playwright install chromium

3. Configure LM Studio

1. Download and install LM Studio
2. Download the MS-FARA 7B model
3. Load the model and start the local server on localhost:1234

4. Install Flask Dependencies (if needed)

pip install flask

🚀 Quick Start

Step 1: Start LM Studio

• Launch LM Studio
• Load the MS-FARA 7B model
• Start the local server (should run on localhost:1234)

Step 2: Start the Onboarding System (Terminal 1)

cd onboarding_system
python app.py
# Running on http://localhost:5001

Step 3: Start the HR System (Terminal 2)

cd hr_system
python app.py
# Running on http://localhost:5002

Step 4: Access the Application

Open your browser and navigate to:

http://localhost:5002

Step 5: Run Onboarding

1. Upload a CSV file with employee data (or use the sample at data/sample_new_hire.csv)
2. Review the parsed employee information
3. Click "Start Onboarding"
4. Watch as the AI automatically:
   • Assigns a seat to the employee
   • Assigns a laptop
   • Assigns monitor(s) based on the requirement
   • Generates notification emails

Project Structure

hr-onboarding-agent/
├── README.md                     # This file
├── hr_system/                    # HR System Flask App (Port 5002)
│   ├── app.py                    # Main Flask application
│   └── templates/
│       └── upload.html           # Upload & monitoring interface
│
├── onboarding_system/            # Onboarding System Flask App (Port 5001)
│   ├── app.py                    # Mock HR system for CUA to control
│   ├── data/
│   │   ├── seats.json            # Available seats data
│   │   └── assets.json           # Available assets data
│   └── templates/
│       └── index.html            # Seat/Asset assignment UI
│
├── hr_onboarding_crew/           # CrewAI Project
│   └── src/hr_onboarding_crew/
│       ├── crew.py               # Crew definition
│       ├── main.py               # Entry point
│       ├── config/
│       │   ├── agents.yaml       # Agent definitions
│       │   └── tasks.yaml        # Task definitions
│       └── tools/
│           ├── cua_tool.py       # Computer Use Agent Tool
│           └── email_tool.py     # Email Sender Tool
│
├── playwright-agent/             # CUA Engine
│   ├── agent.py                  # Main agent loop
│   ├── browser.py                # Playwright browser control
│   ├── prompts.py                # System prompts
│   ├── run_agent.py              # CLI entry point
│   └── config.json               # LM Studio configuration
│
├── data/                         # Input data
│   ├── new_hire.json             # Current employee data
│   └── sample_new_hire.csv       # Sample CSV for testing
│
├── emails/                       # Generated email outputs
│   └── email_*.json
│
└── docs/                         # Documentation
    ├── ARCHITECTURE.md           # Detailed architecture
    ├── CUA_PROMPTING_GUIDE.md    # CUA prompting best practices
    └── QUICKSTART.md             # Quick start guide

🔧 How It Works

1. HR Data Processor Agent

Reads the uploaded CSV data and extracts:

• Employee information (name, department, email)
• AD/SSO account details
• Onboarding requirements (seat preference, equipment needs, monitor count)

2. Onboarding Executor Agent

Uses the CUA Tool to interact with the Onboarding System UI:

Seat Assignment:

1. Finds the first available seat with a green "Available" badge
2. Clicks "Assign Seat"
3. Types the employee's AD username in the modal
4. Confirms the assignment

Asset Assignment (Laptop/Monitor):

1. Navigates to the "Asset Assignment" tab
2. Selects the asset type from the dropdown
3. Assigns the first available asset
4. Repeats for multiple monitors if required

3. Report Generator Agent

Creates and sends notification emails to:

• HR Team: Complete onboarding report
• Manager: New team member notification
• IT Team: Asset assignment confirmation

CUA Prompting Guide

We've documented extensive learnings from developing effective CUA prompts:

| Principle | Description | |-----------|-------------| | Keep it concise | Shorter prompts work better than verbose ones | | Important steps first | Put critical actions at the beginning | | One action per step | Break complex tasks into single actions | | Clear termination | Always specify when to stop | | Visual descriptions | Describe UI elements by their visual appearance |

See docs/CUA_PROMPTING_GUIDE.md for detailed best practices.

Sample CSV Format

employee_id,full_name,email,department,team,job_title,manager_name,manager_email,start_date,office_location,ad_username,equipment_type,monitor_count,seat_zone_preference,floor_preference,special_requirements
EMP-2025-0042,Sarah Johnson,[email protected],Engineering,Backend,Senior Developer,Mike Chen,[email protected],2025.2.1,HQ-Building A,sarahjohnson,Developer Workstation,1,Engineering,3,Standing desk preferred

🔍 Troubleshooting

LM Studio Connection Failed

• Ensure LM Studio is running
• Verify the server is started on localhost:1234
• Check that MS-FARA 7B model is loaded

CUA Not Working

• Verify Onboarding System is running on port 5001
• Check that browser opens in headful (visible) mode
• Review the execution logs for specific errors

Process Stuck

• Click the "Cancel" button in the HR System
• Or press Ctrl+C in the terminal running CrewAI

Stopping the Application

# Press Ctrl+C in each terminal, or:
pkill -f "python.*app.py"
pkill -f "crewai"

📄 License

This project is for educational and demonstration purposes.

Acknowledgments

• CrewAI - Multi-agent orchestration framework
• LM Studio - Local LLM server
• Playwright - Browser automation
• MS-FARA 7B - Vision model for computer use

---

Built with CrewAI + Computer Use Agent (CUA) technology