CrewAI Elasticsearch → GitHub Agent

A containerised CrewAI crew that:

1. Pulls recent log documents from an Elasticsearch cluster (via elasticsearch-py).
2. Analyses them with an LLM-backed SRE Log Analyst agent — identifying error spikes, top failure signatures, and affected services.
3. Publishes the resulting Markdown report (and any small suggested code changes) to a GitHub repository on a feature branch and opens a pull request — via a custom GitHubPushTool built on PyGithub, alongside crewai_tools.GithubSearchTool for repo-aware context.

The project is structured for production use: non-root container, multi-stage build, structured JSON logs, env-only secrets, persistent volumes for logs and artifacts, and a compose file that wires into an existing Elasticsearch cluster (with an optional dev profile for local ES).

Project layout

crewai-es-agent/
├── app/
│   ├── main.py                # container entrypoint
│   ├── crew.py                # agents, tasks, crew assembly
│   ├── settings.py            # env-driven config (fails fast)
│   ├── logging_setup.py       # structlog → stdout + rotating JSONL file
│   ├── es_client.py           # elasticsearch-py wrapper + retry
│   └── tools/
│       ├── es_log_tool.py     # CrewAI BaseTool: query ES
│       └── github_push_tool.py# CrewAI BaseTool: commit + open PR
├── docker/
│   └── entrypoint.sh          # validates env vars, then exec's the app
├── config/
│   └── elastic-ca.crt         # (you provide) CA bundle for private ES TLS
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
├── .dockerignore
└── .env.example

Environment variables

All secrets and tunables are loaded from the environment. Never bake them into the image. Copy .env.example to .env and fill in:

| Variable | Required | Purpose | |---|---|---| | ES_HOSTS | yes | Comma-separated ES URLs, e.g. https://es.internal:9200. | | ES_API_KEY | one-of | Preferred: API key auth. | | ES_USERNAME / ES_PASSWORD | one-of | Basic auth alternative. | | ES_VERIFY_CERTS | no | true/false. Default true. | | ES_CA_CERTS | no | In-container path to a CA bundle (mount via compose). | | ES_INDEX_PATTERN | no | Default logs-*. | | ES_LOOKBACK_MINUTES | no | Default 60. | | ES_MAX_HITS | no | Hard cap on docs fetched per query. Default 1000. | | GITHUB_TOKEN | yes | PAT (repo scope) or fine-grained token with Contents: RW and Pull requests: RW. | | GITHUB_REPO | yes | owner/name. | | GITHUB_BASE_BRANCH | no | Default main. | | GITHUB_TARGET_BRANCH | no | Default crewai/log-analysis. Auto-created if absent. | | GITHUB_OPEN_PR | no | true/false. Default true. | | GIT_AUTHOR_NAME / GIT_AUTHOR_EMAIL | no | Commit metadata. | | LLM_PROVIDER | no | openai (default) or anthropic. | | OPENAI_API_KEY / OPENAI_MODEL | conditional | Needed when provider is OpenAI. | | ANTHROPIC_API_KEY / ANTHROPIC_MODEL | conditional | Needed when provider is Anthropic. | | LOG_LEVEL | no | Default INFO. | | AGENT_LOG_DIR | no | Default /var/log/crewai-agent. | | AGENT_ARTIFACTS_DIR | no | Default /var/lib/crewai-agent/artifacts. |

The docker/entrypoint.sh script validates the critical ones at container start and fails with exit code 78 (EX_CONFIG) if any are missing — so misconfigurations surface immediately instead of mid-run.

Production secret handling

.env is for local use. For real deployments prefer:

• Docker Swarm / Kubernetes: mount each secret as a file and read it via an init-shim that exports it, or use a sidecar like vault-agent / external-secrets to project them as env vars.
• Plain compose: pair with docker swarm secrets or pass via your CI/CD secret store (--env-file <(your-secret-fetch>)).
• Avoid docker inspect-able env vars on shared hosts.

Persistent storage

Two named volumes are declared:

| Volume | Mount point | Contents | Why it's persistent | |---|---|---|---| | agent-logs | /var/log/crewai-agent | agent.jsonl (rotated, 50 MB × 10) | Auditable run history independent of docker logs. | | agent-artifacts | /var/lib/crewai-agent/artifacts | Per-run JSON with crew output + token usage | Replay / debugging after a container exits. |

Both are declared with driver: local in docker-compose.yml. To back them with NFS / EBS / Ceph, replace the driver block — the agent doesn't care where the bytes live, only that the paths are writable.

The image itself is launched read_only: true; only the two volumes and a tmpfs at /tmp are writable, which dramatically reduces blast radius if the LLM or a tool misbehaves.

Connecting to an existing Elasticsearch cluster

The compose file expects an external Docker network called elastic-net — the same network your ES cluster is on. This is the cleanest way to give the agent direct, name-resolvable access to ES without exposing it to the host.

# One-time: create the shared network if your ES stack didn't already
docker network create elastic-net

# Then start the agent
cp .env.example .env
$EDITOR .env

# (Optional) drop your cluster CA into ./config/elastic-ca.crt
docker compose up -d --build agent
docker compose logs -f agent

If your ES cluster lives on a different Docker network, either:

• attach it to elastic-net as well (docker network connect elastic-net <es-container>), or
• change networks: in docker-compose.yml to reference its network as external: true.

If ES is reachable only over the host network (e.g. it's a managed service like Elastic Cloud), you can drop the networks: block on the agent service and set ES_HOSTS=https://your-cluster.es.io:9243 in .env.

Local development with a throwaway ES

docker compose --profile dev-es up -d
# ES is on http://localhost:9200, user `elastic`, password from $ELASTIC_PASSWORD

Update .env for dev:

ES_HOSTS=http://elasticsearch:9200
ES_USERNAME=elastic
ES_PASSWORD=changeme
ES_VERIFY_CERTS=false

Running the agent

Single run (recommended pattern — schedule it externally):

docker compose run --rm agent

Background long-running mode (e.g. you wrap an internal scheduler around it):

docker compose up -d agent

Kubernetes: use a CronJob with restartPolicy: OnFailure, mount the two volumes as PersistentVolumeClaims, and pull all env vars from a Secret.

How the GitHub push works

crewai_tools ships GithubSearchTool, but that tool is read-only (RAG over repo content). For actually pushing analysis output or code changes back to a repo, this project includes a custom GitHubPushTool (in app/tools/github_push_tool.py) that:

1. Authenticates with GITHUB_TOKEN via PyGithub.
2. Ensures GITHUB_TARGET_BRANCH exists (creates it from GITHUB_BASE_BRANCH if not).
3. Upserts each provided file (create_file / update_file with the existing blob SHA).
4. Optionally opens a PR back to GITHUB_BASE_BRANCH.

The Publisher agent calls it through the standard CrewAI tool-call interface — the LLM produces a structured argument set matching GitHubPushInput, and the tool executes the GitHub API calls. This is the supported extension pattern for self-hosted CrewAI deployments; if you adopt CrewAI AMP / Enterprise you can swap this for the managed GitHub integration with OAuth.

Operational notes

• Idempotency: re-running the same day reuses the same target branch and updates the same report file; the PR is created once and silently skipped thereafter.
• Retries: ES queries are wrapped in tenacity (3 attempts, exponential backoff). GitHub calls are not retried — they're usually deterministic failures (auth, permissions) where a retry just wastes credits.
• Cost ceiling: keep ES_MAX_HITS tight and choose a small OPENAI_MODEL for routine runs; escalate manually for incident investigations.
• Healthcheck: the container's Docker healthcheck only verifies that the Python package loads — it does not dial ES or GitHub, because we don't want a transient outage to mark the container unhealthy and break orchestrator semantics.
• Logs: tail with docker compose exec agent tail -f /var/log/crewai-agent/agent.jsonl or ingest the JSONL into the same ES cluster you're analysing (eat your own dog food).

Local smoke test (no Docker)

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
export $(grep -v '^#' .env | xargs)
python -m app.main