# docs-copilot: the AI Engineering course project

A documentation copilot for this website. It answers questions about every lesson on the site, with
**verified citations**, over an **HTTP API** and as an **MCP server**. Every request is **traced with
OpenTelemetry** into Grafana (and optionally Langfuse), and it is **evaluated** against a golden question set.
The walkthrough is `ai-engineering/21-project.html`.

```bash
cd ai-engineering/project
python demo.py                          # offline end to end: index, answers, trace, metrics, eval (~12 s)
python -m pytest                        # 23 tests (+1 skipped unless the MCP SDK is installed)
python -m copilot.evaluate              # golden-set report
uvicorn copilot.api:app --port 8000     # the HTTP API: http://localhost:8000/docs
docker compose up --build               # API + OTel Collector + Grafana/Tempo/Prometheus/Loki
```

If your global Python has broken pytest plugins, run `PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python -m pytest`
or use a virtual environment.

## Modes

| Setting | Effect |
|---------|--------|
| nothing set | offline: `ExtractiveModel` (quotes the best-matching source sentences, **not an LLM**), in-memory telemetry |
| `OPENAI_API_KEY` | real answers from `gpt-5.6-luna` via the Responses API (`COPILOT_MODEL` to change) |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | spans and metrics exported over OTLP (the compose file sets this) |
| `LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY` / `LANGFUSE_BASE_URL` | `/feedback` also stores a Langfuse score; `python -m copilot.evaluate --langfuse` runs an experiment |
| `COLLECTOR_CONFIG=collector-langfuse.yaml` + `LANGFUSE_AUTH` | the Collector also sends every trace to Langfuse |

## Layout

| File | Lessons | What it does |
|------|---------|--------------|
| `copilot/ingest.py` | 07 | HTML → clean, size-capped chunks with metadata; drops quizzes and example output |
| `copilot/index.py` | 06, 08 | BM25 + dense vectors (offline LSA, or OpenAI embeddings) fused with RRF; course filter |
| `copilot/answer.py` | 04, 09, 17 | retrieve → escaped prompt → model → citation check; GenAI-convention spans and metrics |
| `copilot/guardrails.py` | 20 | injection tripwire, PII redaction (Luhn-checked cards), per-user token bucket |
| `copilot/telemetry.py` | 16, 17 | OTel providers (OTLP or in-memory), GenAI histograms with the spec's buckets, app counters |
| `copilot/api.py` | 16, 20 | FastAPI: `/ask`, `/search`, `/feedback`, `/healthz`; continues incoming `traceparent` |
| `copilot/mcp_server.py` | 12–14 | the same copilot as an MCP server (SDK v2): `search_docs`, `ask`, `lesson://` resource, `explain` prompt |
| `copilot/evaluate.py` | 10, 18 | hit@5, MRR, grounded, cites-expected; evaluators in Langfuse's signature |
| `eval/golden.jsonl` | 10 | 41 questions with the lessons that answer them |
| `otel/*.yaml`, `grafana/*` | 19 | Collector pipelines (with redaction) and a provisioned "Docs copilot" dashboard |

## What was verified, and what was not

* **Run and tested here:** ingestion, retrieval, the pipeline with the offline model, guardrails, telemetry
  (in-memory exporters), the FastAPI app (TestClient and a real uvicorn server), the evaluation.
* **Written against current docs but not run here:** the MCP server (needs `pip install "mcp[cli]"`), the
  Langfuse paths (need `langfuse` and an account), the Docker Compose stack (Docker was not running), and
  the Grafana dashboard queries. Metric names in the dashboard assume Prometheus's default OTLP name
  translation; check them in Grafana's metric browser if a panel is empty.