Last Friday I had a shower thought about evals. An eval is a repeatable experiment on an LLM system: fixed inputs, one or more variants under test, and a scoring rule that turns outputs into numbers you can compare. That’s what Claude tells me anyway. evaling can help you if you’ve ever changed a prompt and wondered whether it actually helped.
I quickly jotted down some notes on my phone since ideas come and go quickly, and started a requirements chat with Claude before work. This turned into a pretty detailed markdown file and I sent the agent on its way implementing. That night I had a repo with docs, tests, and was able to run a test eval against Sonnet. I just released that on PyPI.
I think evaling is pretty neat. You configure four things:
– cases — the inputs, held constant so comparisons are fair
– variants — the prompts being compared
– models — what runs them
– scorecard — how outputs become scores
Running evaling produces the matrix: variants × models × cases, one cell each. That’s the unit — one prompt, one model, one case, one score. And a resulting report you can use to choose the right prompt variant and model for your needs.
You can specify a command provider, meaning the thing under test doesn’t have to be a model — it can be a RAG pipeline, an agent, any system that takes text and returns text. There’s an example here.
evaling can be driven via CLI or MCP, so you can kick off evals manually, using a script, or even via an agent. You can provide cases via text files, or via a pageable source you write in Python. The docs are tested — YAML examples are validated against the real config schema and the worked examples run end to end on every commit, so a stale example fails the build. On spend: --max-cost sets a hard ceiling — evaling stops issuing calls when it’s reached, rather than telling you afterwards what you spent. That said, this is pretty new and I’ve only tried some simple evals so it would benefit from your use and feedback.
Please give evaling a try and let me know what you think. You can get started with:
uv tool install evaling
evaling init && evaling run # works offline, no API key
Amro's Blog 