Know what your AI gets right before you rely on it.
AI evaluation is testing a model or AI feature against the job it is meant to do, before people depend on it. Leemer Group builds a test suite around your use case, runs it against the model or product, and reports what works, where it fails and how you would notice.
AI evaluation & assurance
Independent tests for an AI feature before you rely on it: what it gets right, where it fails and how you would know.
We build the test suite around your use case, run it against the model or product, and write up the results in plain language. The aim is a claim you can defend, not a score to quote.
- Task-specific test sets and held-out checks
- Failure analysis and red-teaming
- Model and vendor comparison on your own data
- A written report with evidence, limits and next steps
What is included in an AI evaluation?
Task-specific test sets with held-out checks, failure analysis and red-teaming, comparison of models or vendors on your own data, and a written report with the evidence, its limits and recommended next steps.
How is this different from a benchmark score?
Public benchmarks measure general ability. An evaluation measures your task, with your data and your definition of a good answer. The aim is a claim you can defend, not a number to quote.
Where does this experience come from?
Leemer Labs keeps model claims tied to inspectable evidence through BornBench and held-out gates, and Critique.sh is built around independent review of AI-written code. Client evaluations use the same discipline. The team has also tested models from Z.ai, Moonshot AI and Xiaomi and products from YC- and a16z-backed startups before their public launch.
How an engagement works.
- 01
Brief
A conversation about the problem, the people affected and what a good result looks like. You can draft the idea first in Build Brief.
- 02
Scope
A written scope: what we will build or test, what we will not, and how we will agree it is done.
- 03
Build
Focused work in short cycles. You see working software or evidence early, not a reveal at the end.
- 04
Handover
Code, documents and the reasoning behind decisions, in a form your team can run, change and own.
Questions, answered.
Can you compare AI vendors for us?
- Yes. We run the candidate models or products on the same task-specific tests using your own data, and report the differences with the evidence behind them.
Is this an EU AI Act conformity assessment?
- No. We provide technical testing and evidence about how a system behaves. Legal interpretation and formal conformity assessment belong with qualified legal advisers and notified bodies; our reports can be one input to that work.
What do we receive at the end?
- A written report with the test set, results, observed failures, the limits of the evidence and next steps, plus the tests themselves so you can run them again.
How do we start?
- Use the contact page. It opens an email draft with this service selected, which you review and send yourself. No form collects your message on our side.
Related
services.
Leemer Group designs and builds AI assistants, agents and document search for a defined job, with private deployment when your data should stay with you.
AI-assisted engineeringHelp for engineering teams adopting AI coding agents, with review habits that keep the evidence in front of the person deciding.
AI readiness auditA short, written assessment of which processes are worth automating with AI, which are not, and what to do first.
Tell us what
you’re working on.
Describe the problem and where you are now. We’ll reply with questions, not a sales script. Enquiries open as an email draft that you review and send yourself.