Agentic Customer Support System

A personal educational AI project demonstrating an agentic customer support workflow using CrewAI, LangChain, RAG, FAISS, reranking, web search, Gmail, Google Sheets, and OpenAI.

The system demonstrates how AI agents can classify customer support emails, retrieve information from a local knowledge base, validate generated answers, use web search when additional information is required, generate professional customer responses, and route low-confidence cases for Human-in-the-Loop review.

Note: Slack is used only as an example SaaS product for learning and demonstration purposes. This project is not affiliated with or endorsed by Slack or Salesforce.

---

Disclaimer

This is a personal educational project created solely for learning, experimentation, and portfolio demonstration purposes.

Slack is used in this project only as an example SaaS product to demonstrate how an AI-powered customer support system can work with RAG, multi-agent workflows, web search, Gmail automation, and Human-in-the-Loop (HITL).

This project is not affiliated with, associated with, authorized by, endorsed by, or officially connected with Slack or Salesforce.

The Slack-related knowledge used in this project is intended only for educational and demonstration purposes. No confidential company data, private customer information, or proprietary internal support data is intended to be included in this repository.

The purpose of this project is to learn and demonstrate technologies such as CrewAI, LangChain, RAG, FAISS, embeddings, reranking, web search, Gmail API, Google Sheets API, and agentic AI workflows.

---

Project Goal

The goal of this project is to build a multi-agent AI customer support workflow using three AI agents.

The system is designed to:

• Read incoming customer emails
• Identify whether an email is a genuine support query
• Search a local knowledge base using RAG
• Retrieve relevant information
• Generate an initial support answer
• Validate the generated answer
• Calculate a confidence score
• Trigger web search when confidence is below 70%
• Revalidate the web-based answer
• Generate a professional customer-friendly email
• Send the response through Gmail
• Log the interaction in Google Sheets
• Route unresolved low-confidence cases for Human-in-the-Loop review

---

System Architecture

Customer Email
      |
      v
Gmail API
      |
      v
Support Query Classification
      |
      +----------------------+
      |                      |
      v                      v
   SUPPORT               NOT SUPPORT
      |                      |
      v                    Ignore
 Agent 1
RAG Support Agent
      |
      v
LangChain RAG Pipeline
      |
      v
Document Loading
      |
      v
Text Chunking
      |
      v
Hugging Face Embeddings
all-MiniLM-L6-v2
      |
      v
FAISS Vector Search
      |
      v
Relevant Chunks
      |
      v
CrossEncoder Reranking
ms-marco-MiniLM-L-6-v2
      |
      v
RAG Answer
      |
      v
 Agent 2
Validator + Web Research Agent
      |
      v
Confidence Score
      |
      +-----------------------------+
      |                             |
      v                             v
Confidence >= 70%              Confidence < 70%
      |                             |
      |                             v
      |                         Web Search
      |                             |
      |                             v
      |                       Improve Answer
      |                             |
      |                             v
      |                         Revalidate
      |                             |
      |                    +--------+--------+
      |                    |                 |
      |                    v                 v
      |                 >= 70%             < 70%
      |                    |                 |
      +--------------------+                 |
      |                                      |
      v                                      v
 Agent 3                              Web Search Again
Customer Response Agent                      |
      |                                      v
      |                                  Revalidate
      |                                      |
      |                              +-------+-------+
      |                              |               |
      |                              v               v
      |                           >= 70%            < 70%
      |                              |               |
      +------------------------------+               |
      |                                              v
      v                                  HUMAN REVIEW REQUIRED
Professional Email                                |
      |                                            v
      v                                      Google Sheets
Gmail Reply
      |
      v
Google Sheets
      |
      v
AUTO RESPONDED

---

AI Agents

The project uses three CrewAI agents.

---

Agent 1 - RAG Support Agent

The first agent acts as the primary knowledge-based customer support agent.

Responsibilities

• Understand the customer's support query
• Search the local support knowledge base
• Retrieve relevant information
• Generate an initial support answer
• Use retrieved information instead of relying only on the LLM's knowledge
• Avoid inventing information when the knowledge base is insufficient

Agent 1 has access to the custom:

Search Slack Knowledge Base

tool.

This tool connects CrewAI with the RAG pipeline implemented in rag.py.

---

Agent 2 - Validator and Web Research Agent

The second agent acts as both the:

• Answer Validator
• Web Research Agent

This allows the project to maintain a simple three-agent architecture.

Validation Responsibilities

Agent 2 evaluates the RAG answer based on:

1. Relevance
2. Accuracy
3. Completeness
4. Usefulness

It then generates a confidence score between:

0 - 100

Confidence Routing

If:

Confidence >= 70%

the answer can continue to the customer response stage.

If:

Confidence < 70%

web research is triggered.

---

Web Search Fallback

When the RAG answer does not reach the required confidence threshold, Agent 2 uses web search to find additional information.

The project uses:

SerperDevTool

for web search.

For Slack-related questions, the agent is instructed to prioritize:

• Official Slack documentation
• Reliable sources
• Information directly related to the customer's question

The web-search answer is intentionally kept concise and focused on customer support.

The system currently allows a maximum of:

2 web-search attempts

After each web search, the improved answer is validated again.

---

Agent 3 - Customer Response Agent

The third agent converts the final validated answer into a professional customer-support email.

Responsibilities

• Preserve the factual meaning of the validated answer
• Create a formal response
• Keep the response polite
• Keep the response customer-friendly
• Keep the response concise
• Avoid unnecessary technical jargon
• Avoid adding unvalidated facts
• Avoid inventing troubleshooting steps

The customer email address is handled by the Python workflow rather than being selected or generated by the AI agent.

---

RAG Architecture

The project uses Retrieval-Augmented Generation (RAG) to answer questions using a local knowledge base.

The RAG pipeline is implemented in:

rag.py

The general workflow is:

Knowledge Base Documents
        |
        v
Document Loading
        |
        v
Text Splitting
        |
        v
Embeddings
        |
        v
FAISS Vector Store
        |
        v
Similarity Search
        |
        v
CrossEncoder Reranking
        |
        v
Best Context
        |
        v
CrewAI Agent

---

Knowledge Base

The project can work with multiple knowledge-base documents.

Example:

data/
├── slack_accounts.txt
├── slack_billing.txt
├── slack_integrations.txt
└── slack_notifications.txt

The documents used in the public repository should contain only sample, educational, or publicly available information.

No confidential company or customer information should be included.

---

LangChain

LangChain is used to build and manage the RAG pipeline.

It helps with tasks such as:

• Loading documents
• Splitting documents into smaller chunks
• Creating embeddings
• Connecting embeddings with FAISS
• Performing similarity-based retrieval

This keeps the RAG implementation modular and makes it easier to extend the project to additional document types in the future.

---

Text Chunking

Large documents are divided into smaller pieces before embeddings are created.

This allows the retrieval system to search smaller relevant sections instead of returning an entire document.

Conceptually:

Large Document
      |
      v
Text Splitter
      |
      +--> Chunk 1
      |
      +--> Chunk 2
      |
      +--> Chunk 3
      |
      +--> Chunk 4

These chunks are then converted into embeddings.

---

Embedding Model

The project uses the Hugging Face Sentence Transformer model:

sentence-transformers/all-MiniLM-L6-v2

The embedding model converts text into numerical vector representations.

Conceptually:

"How do I reset my password?"
            |
            v
      Embedding Model
            |
            v
 [0.12, -0.42, 0.81, ...]

These vectors allow FAISS to compare the semantic meaning of the customer query with the knowledge-base content.

---

FAISS Vector Search

The project uses:

FAISS

as the vector store and similarity-search system.

FAISS performs the initial fast retrieval.

The workflow is approximately:

Customer Query
      |
      v
Query Embedding
      |
      v
FAISS Search
      |
      v
Top Relevant Chunks

FAISS is useful because vector similarity search is much faster than asking the LLM to read every document for every customer question.

---

CrossEncoder Reranking

Vector similarity alone may not always return the chunks in the best possible order.

Therefore, the project uses a second model for reranking:

cross-encoder/ms-marco-MiniLM-L-6-v2

The CrossEncoder compares:

Customer Query
      +
Retrieved Document Chunk

and produces a relevance score.

The retrieved documents can then be reordered according to their relevance.

The overall retrieval process becomes:

Customer Query
      |
      v
Embedding
      |
      v
FAISS
      |
      v
Candidate Chunks
      |
      v
CrossEncoder
      |
      v
Reranked Chunks
      |
      v
Best Context

This combines:

FAISS
= Fast retrieval

CrossEncoder
= More accurate ranking

---

LLM

The CrewAI agents use:

GPT-4.1 mini

through OpenAI.

The LLM is used for tasks such as:

• Support answer generation
• Email classification
• Answer validation
• Confidence evaluation
• Web-search result interpretation
• Customer-response generation

The embedding model is separate from the LLM.

---

Support Email Classification

Before running the full RAG workflow, the incoming email is classified as:

SUPPORT

or:

NOT_SUPPORT

This prevents the AI support workflow from replying to unrelated emails.

SUPPORT examples

• Login problems
• Billing questions
• Feature questions
• Product errors
• Troubleshooting requests
• Configuration questions
• Product-related assistance

NOT_SUPPORT examples

• Marketing emails
• Newsletters
• OTP messages
• Promotions
• Advertisements
• Automated notifications
• Meeting invitations
• Spam
• Random personal emails

If the email is classified as:

NOT_SUPPORT

no automatic support reply is sent.

---

Gmail Integration

The project integrates with Gmail using the Gmail API.

Gmail is used to:

• Read unread emails
• Retrieve the sender email address
• Retrieve the email subject
• Retrieve the email body
• Retrieve Gmail thread information
• Send customer responses
• Reply within the original Gmail conversation
• Mark processed support emails as read

---

Base64 Encoding

The Gmail API provides email message data using Base64 encoding.

When reading an email:

Gmail Message
      |
      v
Base64 Decode
      |
      v
Readable Email Text

When sending an email:

Customer Response
      |
      v
MIME Email
      |
      v
Base64 Encode
      |
      v
Gmail API

Base64 is therefore used only for transferring email content through the Gmail API.

It is not part of the AI or RAG pipeline.

---

Google Sheets Integration

Google Sheets is used as a simple support interaction log and Human-in-the-Loop queue.

The sheet contains the following columns:

Received Time
Responded Time
Customer Email
Query
Final Answer
Response Time (seconds)
Status

---

Automatic Response

When the final answer is approved:

Status = AUTO RESPONDED

The system:

1. Sends the customer response
2. Records the responded time
3. Calculates response time
4. Stores the support interaction in Google Sheets
5. Marks the original email as read

---

Human-in-the-Loop (HITL)

The project includes a Human-in-the-Loop safety mechanism.

If the RAG answer has confidence below 70%, web search is performed.

If the answer still remains below 70%, another web-search attempt can be performed.

If the answer still fails to reach the required confidence:

HUMAN REVIEW REQUIRED

The system will:

• Not automatically send an email to the customer
• Store the customer query in Google Sheets
• Store the best available AI answer
• Leave Responded Time blank
• Leave Response Time blank
• Mark the case as HUMAN REVIEW REQUIRED

The workflow is:

RAG
 |
 v
Validation
 |
 v
Confidence < 70%
 |
 v
Web Search
 |
 v
Validation
 |
 v
Confidence < 70%
 |
 v
Web Search Attempt 2
 |
 v
Validation
 |
 v
Confidence < 70%
 |
 v
HUMAN REVIEW REQUIRED
 |
 v
Google Sheets

This prevents low-confidence AI responses from being automatically sent to customers.

---

Response Time Tracking

The project records:

Received Time

and:

Responded Time

The difference between these values is used to calculate:

Response Time (seconds)

This provides a simple way to measure how long the automated support workflow takes to respond.

---

Project Structure

Agentic-Customer-Support/
│
├── data/
│   ├── slack_accounts.txt
│   ├── slack_billing.txt
│   ├── slack_integrations.txt
│   └── slack_notifications.txt
│
├── app.py
├── crew.py
├── rag.py
│
├── requirements.txt
├── README.md
├── .gitignore
└── .env.example

The following files may exist locally but must not be uploaded to GitHub:

.env
credentials.json
token.json
vector_store/
logs/
__pycache__/

---

File Responsibilities

app.py

The main entry point of the application.

It:

• Starts the support workflow
• Retrieves unread Gmail messages
• Processes emails one by one
• Displays the final processing result

---

crew.py

Contains the main agentic workflow.

It includes:

• Gmail integration
• Google Sheets integration
• CrewAI tools
• Three AI agents
• Email classification
• RAG answer generation
• Validation
• Confidence routing
• Web-search fallback
• Customer-response generation
• Human-in-the-Loop handling

---

rag.py

Contains the RAG pipeline.

It handles:

• Document loading
• Text chunking
• Hugging Face embeddings
• FAISS vector storage
• Similarity retrieval
• CrossEncoder reranking
• Final context preparation

---

Technologies Used

The project uses:

• Python
• CrewAI
• LangChain
• OpenAI
• GPT-4.1 mini
• Hugging Face Sentence Transformers
• all-MiniLM-L6-v2
• FAISS
• CrossEncoder
• ms-marco-MiniLM-L-6-v2
• Serper
• Gmail API
• Google Sheets API
• Google OAuth 2.0
• python-dotenv

---

Installation

1. Clone the Repository

git clone <your-repository-url>

Move into the project directory:

cd Agentic-Customer-Support

---

2. Create a Virtual Environment

Windows

python -m venv venv

Activate it:

venv\Scripts\activate

macOS / Linux

python3 -m venv venv

Activate it:

source venv/bin/activate

---

3. Install Dependencies

pip install -r requirements.txt

---

Environment Variables

Create a file named:

.env

Use:

.env.example

as the reference.

Example:

OPENAI_API_KEY=your_openai_api_key_here
SERPER_API_KEY=your_serper_api_key_here
GOOGLE_SHEET_ID=your_google_sheet_id_here

Never commit your actual .env file to GitHub.

---

Google API Setup

The project requires:

• Gmail API
• Google Sheets API

Google OAuth 2.0 is used for authentication.

Create OAuth credentials using Google Cloud Console and download the credentials file.

Save it locally as:

credentials.json

On the first successful authentication, the application creates:

token.json

These files must remain private.

Do not upload:

credentials.json
token.json

to GitHub.

---

Google Sheet Setup

Create a Google Sheet with these columns:

Received Time
Responded Time
Customer Email
Query
Final Answer
Response Time (seconds)
Status

Copy the spreadsheet ID and add it to your .env:

GOOGLE_SHEET_ID=your_google_sheet_id_here

The Received Time and Responded Time columns can be formatted in Google Sheets using:

Format
→ Number
→ Date time

---

Running the Project

After configuring the APIs and environment variables, send a test email to the Gmail account connected to the project.

Run:

python app.py

The application checks the currently available unread emails and processes them.

This assignment version is intentionally designed to be run manually when required rather than operating as a continuously running production service.

---

Example Workflow 1 - RAG Answer

Customer sends:

Subject: Password reset

How do I reset my password?

The workflow can be:

Gmail
   |
   v
SUPPORT
   |
   v
RAG Search
   |
   v
Answer
   |
   v
Validation
   |
   v
Confidence 85%
   |
   v
Agent 3
   |
   v
Professional Email
   |
   v
Gmail Reply
   |
   v
Google Sheets

---

Example Workflow 2 - Web Search Fallback

If the local knowledge base does not contain sufficient information:

Customer Question
      |
      v
RAG
      |
      v
Confidence 55%
      |
      v
Web Search
      |
      v
Improved Answer
      |
      v
Revalidation
      |
      v
Confidence 85%
      |
      v
Customer Response

---

Example Workflow 3 - Human Review

If sufficient confidence cannot be achieved:

Customer Question
      |
      v
RAG
      |
      v
Confidence < 70%
      |
      v
Web Search
      |
      v
Confidence < 70%
      |
      v
Second Web Search
      |
      v
Confidence < 70%
      |
      v
HUMAN REVIEW REQUIRED
      |
      v
Google Sheets

No automatic customer response is sent in this case.

---

Security

Sensitive credentials must never be stored directly in the source code or committed to GitHub.

The following files are excluded using .gitignore:

.env
credentials.json
token.json

Generated files and folders are also excluded:

__pycache__/
vector_store/
logs/

The public repository contains:

.env.example

to show which environment variables are required without exposing actual credentials.

---

Learning Objectives

This project was developed to practice and understand:

• Generative AI
• Agentic AI
• AI agents
• Multi-agent workflows
• CrewAI
• Retrieval-Augmented Generation
• LangChain
• Document chunking
• Embeddings
• Vector databases
• FAISS
• Semantic retrieval
• Reranking
• CrossEncoder models
• LLM validation
• Confidence-based routing
• Web-search fallback
• Human-in-the-Loop
• Gmail API integration
• Google Sheets API integration
• OAuth authentication
• Customer support automation

---

Future Improvements

As this is a learning project, several features could be explored in the future:

• Dedicated Human-in-the-Loop dashboard
• Support ticket integration
• Conversation history and memory
• Multiple knowledge bases
• Additional SaaS products
• Source citations in generated answers
• More advanced confidence evaluation
• Evaluation datasets for RAG quality
• Automated RAG testing
• FastAPI backend
• Event-driven email processing
• Cloud deployment
• Monitoring and analytics

These are possible future improvements and are not required for the current educational assignment.

---

Educational Purpose

This repository represents a personal learning project and is not intended to represent an official Slack customer-support system.

Slack is used only as an example SaaS product to demonstrate the technical architecture of an AI-powered customer-support workflow.

The main purpose of the project is to gain practical experience building an end-to-end agentic AI application involving:

Email
  +
RAG
  +
AI Agents
  +
Validation
  +
Web Search
  +
Human Review
  +
Customer Response

All product names and trademarks belong to their respective owners.