Autonomous Portfolio Rebalancing Agent

A multi-agent portfolio rebalancing system combining CVXPY convex optimization with CrewAI LLM orchestration and XGBoost/SHAP/LIME explainability reporting.

The optimizer solves a quadratic program minimizing tracking error (w − w_target)ᵀ Σ (w − w_target) subject to simplex, long-only, and L1 turnover constraints, using OSQP with ECOS fallback. Three sequential CrewAI agents handle drift analysis, constrained trade generation, and compliance reporting. A 5-tier intervention ladder with a VIX > 40 kill-switch escalates high-risk conditions to human advisors. An XGBoost surrogate model generates SHAP feature attributions and LIME counterfactuals for audit reports. HIFO tax-lot accounting and wash-sale blocking provide tax-aware trade execution.

---

Results

All numbers below are reproducible by running the scripts in this repository.

Drift scoring throughput — benchmark_drift.py, seeded (rng(42)) and deterministic:

| Portfolios | Time (median of 3 runs) | Throughput | |---|---|---| | 50,000 | ~38–41 ms across multiple runs | ~1.2–1.3M portfolios/sec |

Exact timing is hardware-dependent. Run benchmark_drift.py to measure on a given machine.

Real-data backtest — run_real_backtest.py, SPY/AGG/GLD/BIL via yfinance, 15 bps transaction costs, Balanced-profile targets (50/30/12/8):

| Window | Events (Calendar → Optimizer) | Event Reduction | Turnover Reduction | Sharpe (Cal / Opt) | |---|---|---|---|---| | 2015–2019 | 19 → 5 | 73.7% | 39.0% | 1.14 / 1.13 | | 2018–2022 | 19 → 9 | 52.6% | 38.5% | 0.52 / 0.50 | | 2021–2024 | 15 → 6 | 60.0% | 13.0% | 0.79 / 0.81 |

Rebalancing events reduced 53–74% across all three windows at comparable risk-adjusted returns. Turnover reduction of up to 39% (39.0% and 38.5% in two of three windows; 13.0% in the third). The 2021–2024 window shows weaker turnover improvement because the 2022 drawdown followed by a narrow equity rally produced fewer opportunities for the drift-triggered optimizer to improve on calendar rebalancing.

---

Pipeline

graph TD
    MD[Market Data] --> DC[Drift Calculator<br/>vectorised pandas]
    CP[Client Portfolios<br/>50k × 5 risk categories] --> DC
    DC --> TE[Trigger Evaluator<br/>threshold / calendar / event]
    TE --> KS{Kill-Switch<br/>VIX > 40}
    KS -->|Halt| HE[Human Advisor<br/>Escalation]
    KS -->|Safe| CREW

    subgraph CREW [CrewAI Sequential Pipeline]
        A1[Portfolio Analyst] --> A2[Quant Optimizer]
        A2 --> A3[Compliance Officer]
    end

    OPT[CVXPY Engine<br/>QP: min tracking error<br/>s.t. turnover · long-only] <--> A2
    TAX[HIFO Tax-Lot Manager<br/>wash-sale blocking] <--> A2
    XAI[XGBoost Surrogate<br/>SHAP + LIME] <--> A3

    A3 --> OUT[Trade List +<br/>Audit Report]

---

Repository Structure

Portfolio_Rebalancing_Agent/
├── src/
│   ├── engine/                        # Drift calculation and trigger evaluation
│   │   ├── drift_calculator.py        # Vectorised pandas RMSD/SAD drift scoring
│   │   ├── triggers.py                # Threshold, calendar, and event triggers
│   │   ├── threshold_manager.py       # Risk-category drift bands from config
│   │   └── drift_monitor.py           # Batch drift monitoring service
│   ├── optimiser/                     # Constrained trade generation
│   │   ├── portfolio_optimiser.py     # CVXPY QP solver (OSQP / ECOS fallback)
│   │   ├── constraint_manager.py      # Turnover and liquidity limits from thresholds.yaml
│   │   ├── tax_lot_manager.py         # HIFO tax-lot accounting
│   │   ├── tax_optimiser.py           # Wash-sale blocking and tax impact calculation
│   │   ├── tax_harvesting_scanner.py  # Proactive tax-loss harvesting identification
│   │   ├── cost_estimator.py          # Brokerage + STT cost model
│   │   ├── liquidity_scorer.py        # ADV-based liquidity constraints
│   │   └── trade_list_generator.py    # Weight-to-share conversion
│   ├── orchestration/                 # CrewAI multi-agent pipeline
│   │   ├── crew_definition.py         # 3 sequential agents: Analyst, Optimizer, Compliance
│   │   ├── agent_tools.py             # @tool wrappers exposing engines to LLM agents
│   │   ├── orchestrator.py            # Top-level orchestration logic
│   │   ├── pipeline_runner.py         # End-to-end pipeline execution
│   │   └── workflow_state.py          # Pipeline state management
│   ├── explainability/                # Audit and compliance reporting
│   │   ├── surrogate_model.py         # XGBoost classifier (rebalance / no-rebalance)
│   │   ├── shap_lime_explainer.py     # SHAP attributions + LIME counterfactuals
│   │   ├── generators.py              # Markdown report generation via LLM
│   │   ├── templates.py               # Report templates by audience level
│   │   └── qa_checker.py              # Numerical consistency validation
│   ├── human_in_loop/                 # Intervention and kill-switch logic
│   │   ├── intervention_classifier.py # 5-tier ladder + VIX kill-switch
│   │   └── override_capture.py        # Advisor override logging with audit trail
│   ├── compliance/                    # Regulatory audit modules
│   │   ├── compliance_auditor.py      # Automated quarterly audit sampling
│   │   └── regulatory_reporter.py     # SEBI-compliant report generation
│   ├── dashboard/                     # Streamlit UI (5 views)
│   │   ├── app.py                     # Portfolio Overview, Activity, Analytics, Explainability, Health
│   │   └── mock_data_generator.py     # Demonstration data for the dashboard
│   ├── backtesting/                   # Strategy comparison framework
│   │   ├── backtest_engine.py         # Historical replay engine
│   │   ├── strategy_comparator.py     # Multi-strategy comparison runner
│   │   └── performance_analyser.py    # Sharpe, drawdown, turnover metrics
│   ├── simulation/                    # Market scenario simulation
│   │   └── scenario_runner.py         # Extreme-scenario stress tests
│   └── data/                          # Data models and generators
│       ├── models.py                  # Pydantic: Portfolio, TaxLot, ClientProfile, etc.
│       └── portfolio_generator.py     # Synthetic portfolio generation (50k portfolios)
├── config/
│   ├── thresholds.yaml                # Turnover, cost, tax, and liquidity parameters
│   ├── risk_categories.yaml           # 5 risk profiles with target allocations and drift bands
│   └── default.yaml                   # Logging and simulation settings
├── tests/                             # Pytest suite covering all modules
├── reports/
│   └── backtests/                     # Backtest output reports
├── benchmark_drift.py                 # Reproducible drift throughput benchmark
├── run_real_backtest.py               # Three-window real-data backtest (yfinance)
├── generate_report.py                 # Report generation entry point
├── requirements.txt
├── .env.example                       # Placeholder keys only
└── .github/workflows/ci.yml           # CI workflow (pytest + scenario runner)

---

Quickstart

git clone https://github.com/SatvikJ01/Portfolio_Rebalancing_Agent.git
cd Portfolio_Rebalancing_Agent
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
# Add your OPENAI_API_KEY to .env (required for CrewAI agent execution)

Reproduce the drift benchmark

python benchmark_drift.py

Scores 50,000 portfolios across 5 risk categories. Output reports median time and throughput. The RNG seed is fixed at 42 for deterministic portfolio generation.

Reproduce the real-data backtest

python run_real_backtest.py

Downloads SPY/AGG/GLD/BIL daily close prices from yfinance and runs buy-and-hold, calendar (quarterly), and CVXPY-optimizer strategies across three windows: 2015–2019, 2018–2022, and 2021–2024. Requires an internet connection for the initial data download.

Run the test suite

pytest tests/ -v

Launch the Streamlit dashboard

streamlit run src/dashboard/app.py

The dashboard displays 5 views: Portfolio Overview (drift heatmap), Rebalancing Activity, Performance Analytics, Explainability Centre, and System Health. It uses mock data generators for demonstration.

---

Technical Detail

<details> <summary>Optimization Formulation (CVXPY)</summary>

The optimizer (src/optimiser/portfolio_optimiser.py) solves a quadratic program using CVXPY:

Objective — minimize tracking error:

min  (w - w_target)ᵀ Σ (w - w_target)

where w is the decision variable (new portfolio weights), w_target is the strategic target allocation, and Σ is the asset return covariance matrix.

Constraints:

| Constraint | Formulation | Source | |---|---|---| | Fully invested (simplex) | sum(w) = 1.0 | Hard constraint | | Long-only | w >= 0 | Hard constraint | | Turnover limit | 0.5 * \|\|w - w_current\|\|₁ <= 0.20 | config/thresholds.yaml → constraints.max_turnover | | Liquidity (optional) | \|w_i - w_current_i\| <= max_weight_change_i | ADV-based bounds from ConstraintManager |

The 20% turnover constraint caps one-way portfolio turnover: at most 20% of portfolio value can change sides in a single rebalancing event.

Solver strategy: OSQP (first-order QP solver, fast for moderate-size problems) with automatic fallback to ECOS if OSQP fails to converge. Post-solve, weights are clipped to non-negative and renormalized.

Design rationale (ADR 2): CVXPY provides mathematically guaranteed optimal solutions for convex problems. Heuristic or gradient-descent approaches cannot guarantee constraint satisfaction, which is critical for a system that must enforce regulatory turnover limits.

</details> <details> <summary>Agent Pipeline (CrewAI)</summary>

The system uses three sequential CrewAI agents (src/orchestration/crew_definition.py), each with a narrow scope and dedicated tools:

| Agent | Role | Tool | Scope | |---|---|---|---| | Portfolio Analyst | Evaluate drift and determine if rebalancing is needed | calculate_drift_tool | Read-only: outputs "Proceed" or "Stop" | | Quantitative Optimizer | Generate constraint-satisfying trade list | generate_optimized_trades_tool | Calls CVXPY engine; cannot execute trades | | Compliance Officer | Produce audit-ready explanation report | generate_explanations_tool | Calls XGBoost/SHAP/LIME engine |

Why three agents instead of one: Each agent receives a strict prompt scoped to a single task. If the Optimizer agent produces an infeasible trade suggestion, the CVXPY tool returns an error rather than executing a bad trade — the agent must retry with corrected inputs. This architecture converts LLM hallucination from a silent failure mode into an explicit, recoverable error.

Sequential processing (Process.sequential): The output of each agent becomes input to the next. There is no parallel execution or dynamic delegation — the pipeline is deterministic in structure, even though individual agent responses are non-deterministic.

Design rationale (ADR 1): CrewAI was chosen over raw LangChain prompt chaining because it natively supports role-based agent definitions and sequential task handoff with less boilerplate. The LLM model is configurable via the llm_model parameter in RebalancingCrew (default: gpt-4o-mini).

</details> <details> <summary>Explainability Layer (XGBoost + SHAP/LIME)</summary>

The explainability pipeline (src/explainability/) provides regulator-grade decision audit reports:

XGBoost surrogate model (surrogate_model.py): A gradient-boosted classifier that predicts rebalance/no-rebalance decisions from five features:

• drift_magnitude_pct — current portfolio drift
• vix_level — market volatility index
• days_since_last_rebalance — time since last rebalancing event
• client_risk_score — client risk category (1–5)
• sector_concentration_pct — maximum sector weight

The surrogate is currently trained on synthetic data generated via a logistic function that approximates the agent's triggering logic. This enables SHAP/LIME analysis without requiring a large corpus of actual agent decisions.

SHAP (shap_lime_explainer.py): Tree SHAP values computed via shap.TreeExplainer provide global feature importance — the exact additive contribution of each feature to a given decision. Waterfall plots are saved to reports/plots/.

LIME (shap_lime_explainer.py): Local Interpretable Model-agnostic Explanations generate counterfactual statements for individual decisions (e.g., "If drift magnitude had been below 3.2% instead of 4.7%, the agent would not have triggered a rebalance").

Design rationale (ADR 3): The SHAP/LIME hybrid provides both structural global importance (required for aggregate audit) and local counterfactuals (required for individual decision justification). This combination addresses both SEC/SEBI regulatory audit requirements: "why does this system behave this way in general" and "why did it make this specific trade."

</details> <details> <summary>Backtest Methodology</summary>

The real-data backtest (run_real_backtest.py) compares three strategies on historical ETF prices:

Strategies:

| Strategy | Rebalance trigger | Trade target | |---|---|---| | Buy-and-hold | Never | — | | Calendar | Every 63 trading days (~quarterly) | Reset to target weights | | CVXPY optimizer | When max single-asset drift exceeds 3% band | CVXPY-optimized weights using 60-day trailing covariance |

Parameters:

| Parameter | Value | Source | |---|---|---| | Tickers | SPY, AGG, GLD, BIL | Mapped to Equity, Fixed Income, Alternatives, Cash | | Target allocation | 50/30/12/8 | Balanced profile from config/risk_categories.yaml | | Drift band | 3% | Balanced profile drift band | | Transaction cost | 15 bps flat | config/thresholds.yaml: brokerage (5 bps) + STT (10 bps) | | Covariance lookback | 60 trading days | Annualized (×252) sample covariance | | Calendar interval | 63 trading days | ~1 calendar quarter |

Three-window design: The backtest runs across three overlapping windows — 2015–2019 (steady bull market), 2018–2022 (includes COVID crash and recovery), and 2021–2024 (rate-hiking cycle and 2022 drawdown). The multi-window design tests whether findings hold across different market regimes rather than reflecting a single favorable period.

Cost model limitation: The 15 bps flat cost does not include market impact. The config/thresholds.yaml file defines a market_impact_coefficient used by src/optimiser/cost_estimator.py, but the backtest script uses the flat cost for simplicity and reproducibility.

</details> <details> <summary>Tax-Lot Accounting</summary>

The tax module (src/optimiser/tax_lot_manager.py, src/optimiser/tax_optimiser.py) provides HIFO lot selection and wash-sale blocking:

HIFO (Highest In, First Out): When selling shares, lots are sorted by cost basis in descending order. Selling the highest-cost lots first minimizes realized capital gains (or maximizes realized losses for tax-loss harvesting).

Wash-sale blocking: The TaxOptimiser tracks realized losses per (portfolio, ticker, date). If a loss was realized within the past 30 days (configurable via config/thresholds.yaml → tax_and_liquidity.wash_sale_window_days), a repurchase of the same security is blocked to avoid IRS wash-sale disallowance.

Tax impact calculation: Distinguishes short-term gains (held < 365 days, taxed at 30%) from long-term gains (held >= 365 days, taxed at 15%). These rates and thresholds are configurable in config/thresholds.yaml.

Integration: The TaxLotManager is available as a tool to the Quant Optimizer agent and can be invoked during trade generation to select tax-optimal lots for each sell order.

</details> <details> <summary>Intervention and Kill-Switch Logic</summary>

The human-in-the-loop module (src/human_in_loop/intervention_classifier.py) implements a 5-tier intervention ladder:

| Level | Meaning | |---|---| | INFORMATIONAL | No action required; logged for audit | | ADVISORY | Advisor is notified but trade proceeds | | APPROVAL_REQUIRED | Trade is held pending advisor approval | | ESCALATION | Briefing document generated for human review | | HALT | All autonomous trading suspended |

Kill-switch: If VIX exceeds 40.0 (configurable via max_vix parameter, also set in .env.example as VIX_KILL_SWITCH_THRESHOLD), the check_kill_switch method returns True, halting all autonomous rebalancing. A secondary trigger halts on system error rate exceeding 1%.

Trade-level classification: The classify_decision method assigns an intervention tier based on proposed turnover as a fraction of portfolio value. Turnover above max_turnover_pct (default 10%) requires approval; turnover above half that threshold triggers an advisory.

</details>

---

Limitations

• Paper backtest only. This system has not been live-traded. The backtest uses historical close prices without slippage, partial fills, or intraday price variation.
• Flat cost model in backtest. Transaction costs are modeled as 15 bps per trade (brokerage + STT). No market impact model (square-root or otherwise) is applied in the backtest, though a market_impact_coefficient is defined in config/thresholds.yaml and used by src/optimiser/cost_estimator.py.
• Surrogate model trained on synthetic data. The XGBoost surrogate generates its own training data via a logistic function that approximates the agent's triggering logic. The architecture supports training on actual agent decisions, but this has not been implemented with a production-scale decision corpus.
• Mock data in agent tools. The CrewAI agent tools (src/orchestration/agent_tools.py) use a hardcoded PORTFOLIO_DB dictionary for demonstration. Connecting to a live portfolio database requires replacing this data source.
• Dashboard uses mock data. The Streamlit dashboard displays generated demonstration data, not live output from the optimization pipeline.
• yfinance data variability. Adjusted close prices from yfinance may differ slightly between downloads due to corporate action adjustments and data revisions. Minor numerical differences in backtest results are expected across runs on different dates.
• Single-broker cost model. The cost estimator assumes Indian market fee structures (brokerage + STT). Other jurisdictions would require different cost parameters.