Connect Claude via MCP
Plug Claude into Librari Evals over MCP — browse schemas, edit prompts, run benchmarks, and extract documents from a chat, with your own personal key.
Everything this manual describes clicking through — building a schema, filling in ground truth, running a benchmark, reading the scores — you can also just ask for. The Eval Engine ships an MCP (Model Context Protocol) server, and any Claude surface — claude.ai, Claude Desktop, Claude Code — can connect to it and use the eval system on your behalf.
Claude acts as you: your login, your tenants, your permissions. It can't see another tenant's data, can't edit published records you couldn't edit, and everything it does is the same server-checked operation the admin UI would have performed.
What you'll need
- A Librari Evals account assigned to your tenant (the same one you use for
/admin). - One of: claude.ai (Team/Pro with connectors), the Claude Desktop app, or Claude Code in a terminal.
- For the tools that call an LLM (extraction, test runs): a tenant LLM provider key linked to the model deployment you want to use.
Step 1 — Generate your personal key
1. Open your own user record
Sign in at /admin, open Users (under Settings), and click your own email.
2. Enable the key
Tick Enable API Key (for connecting Claude / MCP). A key appears in the field below — labeled as your personal MCP connector key. Copy it.
Step 2 — Connect your Claude
claude.ai / Claude Desktop — add a custom connector and give it this URL (your key is part of the address, since connector UIs can't set headers):
https://evals.thelibrari.com/api/mcp/<your-key>Claude Code — add the server with the key in a header instead:
claude mcp add --transport http eval-engine https://evals.thelibrari.com/api/mcp \
--header "Authorization: Bearer <your-key>"Any other MCP-capable client works the same way: point it at /api/mcp with Authorization: Bearer <your-key>, or use the key-in-URL form when headers aren't an option.
Step 3 — Say hello
Start with something like "run whoami and tell me what you can see." Claude calls the whoami tool, which confirms the key works, names your tenants, and lists every collection it can browse. From there, talk normally:
- "List my request schema types and tell me which versions aren't published yet."
- "Why can't the Contract Dates version publish?" (Claude runs
check_publishand reads you the exact blockers.) - "Show me what schema version X actually sends to the model." (
get_compiled_schema— base prompt, field prompts, enums, the lot.)
What Claude can do
Nineteen tools, grouped by job:
| Area | Tools | In plain words |
|---|---|---|
| Look around | whoami, list_records, get_record, get_compiled_schema | Browse and search everything you can see — schemas, ground truth, results, the model catalog — plus the exact compiled prompt+schema a version sends to the LLM. |
| Build schemas | create_record, update_record, delete_record, create_schema_field, create_base_prompt_version, create_field_prompt_version, assign_field_prompt | Create document types, schemas, fields, and prompts. Version numbers are handled automatically; field naming rules are checked with friendly errors. |
| Publish safely | check_publish, set_status, update_enum_values | Dry-run publish checks, draft → published → archived transitions, and the guided flow for changing an enum's allowed values when ground truth still uses one. |
| Ground truth | upload_ground_truth_document, upsert_ground_truth_response, prepopulate_ground_truth | Upload PDFs (by link or attachment, with optional OCR), record the correct answers, or have a model draft the answers for you to correct. |
| Run benchmarks | list_eligible_test_sets, start_test_run, run_test_document, finalize_test_run, get_test_results, regrade_test_document | The full manual test cycle from a chat: start a run, process documents one by one, get per-field verdicts, and fix grades by hand. |
| Extract | extract_document | "Read this PDF with schema X and model Y" — returns the structured data plus token usage and cost. |
Guardrails worth knowing
- Publishing is checked, not hoped.
set_statusruns the same publish validation the admin UI uses and tells Claude why something can't publish instead of failing. - Published records stay locked. Non-super-admins can't edit them through Claude any more than through the UI; the one exception (enum values) goes through the guided remap flow.
- Deletes are drafts-only. Anything published gets archived instead.
- Extractions cost real money.
extract_document,prepopulate_ground_truth, andrun_test_documenteach make a real LLM call, billed to your tenant's provider key, and report the cost back. A big vision-heavy document can take minutes per call. - One document per call. Test runs process document-by-document — Claude narrates progress instead of disappearing into a long job.
Troubleshooting
| Symptom | What's going on |
|---|---|
| 401 mentioning "this looks like a tenant API key" | You pasted an Extraction API Key. Generate a key on your user record instead (Step 1). |
| "Not found" on a record you know exists | It probably belongs to a tenant you're not in — reads are scoped to your tenants, and other tenants' records look identical to records that don't exist. |
| "Several LLM keys cover that deployment" | Your tenant has more than one provider key linked to that model. Tell Claude which one (tenantLlmApiKeyId), or ask it to list them. |
| "No LLM provider key in your tenant covers that model deployment" | Link a key to the deployment under Tenant LLM API Keys first. |
Where to go next
New to the domain objects Claude will be talking about? Concepts defines them all, and The Workflow shows how they fit together. If you'd rather click than chat, every one of these operations also has a manual page — start with Build a Request Schema.
OCR Large Documents with Async Jobs
Submit big scans to /api/ocr-async, poll a job id, and download an OCRd PDF that would time out the synchronous OCR call.
Manage LLM Provider API Keys
Register your OpenAI, Anthropic, Azure, AWS, or Vertex credentials so manual and scheduled runs can call those providers on your behalf.