The tools landscape

There are dozens of eval tools, and their marketing pages all sound the same. Underneath, they fall into five families that answer different questions. Once you see the families, and see that every tool is the same four boxes with different names, picking one becomes a short decision rather than a week of reading.

A mental model: five families of tools

Every tool in this module implements the pipeline from Module 01: a dataset of inputs, a system under test that produces outputs, a grader that scores them, and a metric that aggregates scores so you can compare runs. What differs is who the tool was built for and therefore which question it answers best. That gives five families.

In one sentence each:

• Research harnesses run standardized tasks against models, often in sandboxes, with a strong focus on reproducibility. Their user is a researcher comparing models, not a product team comparing prompts.
• Developer test and CI tools feel like a test runner. You write cases in a config file or as test functions, run a command, and get pass or fail. They plug into GitHub Actions or pytest.
• Observability plus eval platforms start from tracing: recording every model call, tool call and step of your live application as a tree of spans (a span is one timed unit of work, such as "call the search tool"; a trace is the whole tree for one request). They add datasets, experiments and LLM judges on top, usually with a web UI for the team.
• Metric and judge libraries give you scoring functions (faithfulness, factuality, helpfulness) and nothing else. You call them from whichever harness or platform you use.
• Cloud vendor services are evaluation features inside a model provider's or cloud's console, convenient if you are already committed to that vendor.

The landscape map

Families are useful, but tools blur the edges. A second way to see the market is a map with two axes:

• Horizontal: offline and research ↔ online and production. Offline means you run a fixed dataset before release. Online means you score real traffic after release (see Module 15).
• Vertical: code-first library ↔ hosted platform. A library is something you pip install and run on your own machine or CI. A hosted platform has a server, a database and a team UI (run by a vendor, or by you if it is self-hostable).

Read the map diagonally. The bottom-left is where frontier labs, safety institutes and academics live: free, code-first, reproducible, no production concept at all. The top-right is where product teams live once they have real users: hosted UIs, traces from production, judges running on sampled traffic. The bottom-middle is the developer's inner loop: a CLI or pytest in CI. The top-left is vendor consoles: easy, point-and-click, tied to one provider.

Tool by tool

Each card gives the maker, license, language, core abstractions (the nouns you will type), what it is best at, and how it handles agents, multimodal inputs and LLM judges (Module 13). Snippets are copied from each tool's official docs or README (trimmed where marked). If a feature is not listed, it is because it could not be verified from a primary source, not necessarily because it is missing.

Research harnesses

from inspect_ai import Task, task
from inspect_ai.dataset import FieldSpec, hf_dataset
from inspect_ai.scorer import model_graded_qa
from inspect_ai.solver import generate

@task
def simpleqa():
    return Task(
        dataset=hf_dataset(
            "codelion/SimpleQA-Verified",
            split="train",
            sample_fields=FieldSpec(input="problem", target="answer"),
        ),
        solver=generate(),          # swap for an agent, e.g. react() with tools
        scorer=model_graded_qa(),   # an LLM judge compares output to target
    )

pip install inspect-evals
# run a ready-made benchmark against any model Inspect supports
inspect eval inspect_evals/gpqa_diamond --model <provider/model> --epochs 3 --limit 50
inspect view                       # browse the logs

lm_eval --model hf \
    --model_args pretrained=EleutherAI/gpt-j-6B \
    --tasks hellaswag \
    --device cuda:0 \
    --batch_size 8

helm-run --run-entries mmlu:subject=philosophy,model=openai/gpt2 --suite my-suite --max-eval-instances 10
helm-summarize --suite my-suite
helm-server --suite my-suite   # browse results at http://localhost:8000/

export ANTHROPIC_API_KEY=<YOUR-KEY>
harbor run --dataset terminal-bench@2.0 \
   --agent claude-code \
   --model anthropic/claude-opus-5-5 \
   --n-concurrent 4

Developer test and CI tools

prompts:
  - file://prompt1.txt
  - file://prompt2.txt
providers:
  - openai:gpt-6-luna
  - id: vertex:gemini-3.5-flash
    config:
      region: global
tests:
  - vars:
      language: French
      input: Hello world
    assert:
      - type: llm-rubric
        value: does not describe self as an AI, model, or chatbot

from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams  # older docs: LLMTestCaseParams

def test_case():
    correctness = GEval(
        name="Correctness",
        criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
        evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
        threshold=0.5,
    )
    tc = LLMTestCase(input="What if these shoes don't fit?",
                     actual_output="You have 30 days to get a full refund at no extra cost.",
                     expected_output="We offer a 30-day full refund at no extra costs.")
    assert_test(tc, [correctness])

from pydantic_evals import Case, Dataset

dataset = Dataset(name='capital_quiz', cases=[Case(name='simple_case',
          inputs='What is the capital of France?', expected_output='Paris')])

async def guess_city(question: str) -> str:
    return 'Paris'

dataset.evaluate_sync(guess_city).print()

Observability plus eval platforms

from braintrust import Eval
from autoevals import ExactMatch

Eval(
    "Evaluation quickstart",
    experiment_name="Movie matcher (Python)",
    data=[{"input": "A detective investigates a series of murders based on the seven deadly sins.",
           "expected": "Se7en"}],
    task=task,          # def task(input): -> str (calls the model)
    scores=[ExactMatch],
)

from langsmith import Client
from openevals.llm import create_llm_as_judge
from openevals.prompts import CORRECTNESS_PROMPT

def correctness_evaluator(inputs, outputs, reference_outputs):
    judge = create_llm_as_judge(prompt=CORRECTNESS_PROMPT, model="openai:o3-mini",
                                feedback_key="correctness")
    return judge(inputs=inputs, outputs=outputs, reference_outputs=reference_outputs)

Client().evaluate(target, data="Sample dataset", evaluators=[correctness_evaluator],
                  experiment_prefix="first-eval-in-langsmith", max_concurrency=2)

from langfuse import get_client, Evaluation
langfuse = get_client()

def accuracy_evaluator(*, input, output, expected_output, metadata, **kwargs):
    ok = expected_output and expected_output.lower() in output.lower()
    return Evaluation(name="accuracy", value=1.0 if ok else 0.0)

result = langfuse.run_experiment(name="Geography Quiz", data=local_data,
                                 task=my_task, evaluators=[accuracy_evaluator])
print(result.format())

from phoenix.evals import create_classifier
from phoenix.evals.llm import LLM

llm = LLM(provider="openai", model="gpt-4o")
evaluator = create_classifier(
    name="helpfulness",
    prompt_template="Rate the response to the user query as helpful or not:\n\nQuery: {input}\nResponse: {output}",
    llm=llm,
    choices={"helpful": 1.0, "not_helpful": 0.0},
)
scores = evaluator.evaluate({"input": "How do I reset?", "output": "Go to settings > reset."})

import weave, asyncio
from weave import Evaluation, Model

weave.init('intro-example')

@weave.op()
def match_score1(expected: str, output: dict) -> dict:
    return {'match': expected == output['generated_text']}

class MyModel(Model):
    prompt: str
    @weave.op()
    def predict(self, question: str):
        return {'generated_text': 'Hello, ' + question + self.prompt}

evaluation = Evaluation(dataset=examples, scorers=[match_score1])
asyncio.run(evaluation.evaluate(MyModel(prompt='World')))

from opik import track

@track                       # traces this function
def my_function(input: str) -> str:
    return input

# the evaluate() call as shown in Opik's docs:
evaluation = evaluate(dataset=dataset, task=evaluation_task, scoring_metrics=[Hallucination()])

import mlflow
from mlflow.genai import scorer
from mlflow.genai.scorers import Correctness, Guidelines

@scorer
def is_concise(outputs: str) -> bool:
    return len(outputs.split()) <= 5

eval_dataset = [{"inputs": {"question": "What is the capital of France?"},
                 "expectations": {"expected_response": "Paris"}}]

results = mlflow.genai.evaluate(
    data=eval_dataset,
    predict_fn=qa_predict_fn,     # def qa_predict_fn(question: str) -> str
    scorers=[Correctness(), Guidelines(name="is_english", guidelines="The answer must be in English"), is_concise],
)

Metric and judge libraries

from openai import AsyncOpenAI
from ragas.metrics import DiscreteMetric
from ragas.llms import llm_factory

llm = llm_factory("gpt-4o", client=AsyncOpenAI())
metric = DiscreteMetric(
    name="summary_accuracy",
    allowed_values=["accurate", "inaccurate"],
    prompt="Evaluate if the summary is accurate and captures key information.\nResponse: {response}\nAnswer with only 'accurate' or 'inaccurate'.",
)
score = await metric.ascore(llm=llm, response="The summary of the text is...")
print(score.value, score.reason)

from trulens.core import TruSession, Metric
from trulens.providers.openai import OpenAI
from trulens.apps.app import TruApp

session = TruSession()
provider = OpenAI(model_engine="gpt-4-turbo")
f_groundedness = Metric(
    implementation=provider.groundedness_measure_with_cot_reasons_consider_answerability,
    name="Groundedness")
tru_rag = TruApp(rag, app_name="RAG", feedbacks=[f_groundedness])
with tru_rag as recording:
    rag.query("What is ...?")
session.get_leaderboard()

Cloud vendor services

eval_obj = client.evals.create(
    name="IT Ticket Categorization",
    data_source_config={"type": "custom", "item_schema": {...}, "include_sample_schema": True},
    testing_criteria=[{
        "type": "string_check", "name": "Match output to human label",
        "input": "{{ sample.output_text }}", "operation": "eq",
        "reference": "{{ item.correct_label }}",
    }],
)

# Google GenAI Client (from the python-aiplatform README; model id may be dated)
inference_results = client.evals.run_inference(model="gemini-2.5-flash-preview-05-20", src=prompts_df)
eval_result = client.evals.evaluate(
    dataset=inference_results,
    metrics=[types.Metric(name='exact_match'), types.Metric(name='rouge_l_sum'), types.RubricMetric.TEXT_QUALITY],
)

Beyond the five families: red teaming, transcripts, public numbers

Three more kinds of tool sit next to the five families. They are not rivals to your harness or platform; each answers a question the others do not. Red-teaming scanners ask "how can this be broken?". Transcript analysis tools ask "what is actually happening across thousands of runs?". Third-party measurement asks "what do independent runs say about models I have not tested yet?". The map at the end of this section places them next to the tools you already know.

Red-teaming scanners

The tools above mostly run datasets you wrote. Red-teaming scanners bring the dataset with them: large, curated families of attacks (jailbreaks, prompt injection, encoding tricks, data-leak bait), a way to send them to your model or app, and a detector that decides whether each attack got through. Think of them as an eval tool with an adversarial test generator built in. The concepts behind them (attack success rate, injection, jailbreak robustness) are in Module 17 and Module 25; here is how the two best-known open-source scanners are built.

python -m pip install -U garak
export OPENAI_API_KEY="..."
# run the encoding-based prompt-injection probes against one hosted model
python -m garak --model_type openai --model_name gpt-4o-mini --probes encoding
# flag names change between releases (newer ones also accept --target_type and
# --target_name); check `python -m garak --help` for your installed version

Transcript analysis

Once agent evals produce thousands of long transcripts, reading them all is impossible and reading ten at random misses rare problems. A new kind of tool helps a human search and summarize transcripts at scale. This is error analysis (Module 12) with power tools.

Third-party measurement

Finally, some sources are not tools you run at all: they are organizations that run evals and publish the numbers. They are useful for a shortlist and dangerous as a final answer.

What changed recently

Tool choice is partly a bet on who will still maintain the tool in two years. The last two years saw acquisitions, handovers and one shutdown.

The comparison matrix

Legend: Y first-class, P partial or via integration, – not a focus, ? not verified from a primary source (treat as unknown, not as "no"). Sources are the per-tool cards above and the list at the end.

Same eval, five tools

The fastest way to understand a tool is to write the same tiny eval in it. Here is the spec, in plain English:

• Dataset: two questions, "What is the capital of France?" (expected: Paris) and "What is the capital of Australia?" (expected: Canberra). Australia is a classic trap: models sometimes say Sydney.
• System under test: a model answering the question.
• Grader: an LLM judge that decides whether the answer is correct compared with the expected answer (a model-graded rubric, see Module 13). Where cheap, also a code check that the expected word appears.
• Metric: fraction of samples passing.

1. Inspect

A Python function decorated with @task returns a Task. The dataset can be a plain list of Sample objects. model_graded_qa() asks a judge model whether the output answers the question correctly given the target.

from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import model_graded_qa
from inspect_ai.solver import generate

@task
def capitals():
    return Task(
        dataset=[
            Sample(input="What is the capital of France?", target="Paris"),
            Sample(input="What is the capital of Australia?", target="Canberra"),
        ],
        solver=generate(),         # the system under test: one model call
        scorer=model_graded_qa(),  # LLM judge: is the answer correct vs target?
    )

# Run from a shell (the model is chosen at run time, not in the file):
#   inspect eval capitals.py --model openai/gpt-4o
# then open the log in Inspect View:  inspect view

Notice what is not in the file: the model. Inspect treats the model as a run-time parameter, because its users mostly compare many models on the same task. To turn this into an agent eval you would change one line, the solver, to an agent such as react() with tools and a sandbox.

2. promptfoo

No code at all. The YAML file declares prompts, providers (models) and tests. Every test is run against every prompt × provider combination.

# promptfooconfig.yaml
prompts:
  - "Answer in one short sentence: {{question}}"
providers:
  - openai:gpt-4o
tests:
  - vars:
      question: What is the capital of France?
    assert:
      - type: contains          # cheap code check first
        value: Paris
      - type: llm-rubric        # LLM judge with a plain-English rubric
        value: Correctly states that Paris is the capital of France, with no false claims
  - vars:
      question: What is the capital of Australia?
    assert:
      - type: contains
        value: Canberra
      - type: llm-rubric
        value: Correctly states Canberra, and does not claim Sydney is the capital

# Run:  npx promptfoo eval     (then:  npx promptfoo view)

The grid is the point: add a second prompt and a second provider and you get a 2 × 2 × 2 comparison for free. This is why promptfoo is popular for "which model should we switch to?" questions and for CI diffs on pull requests.

3. DeepEval

Each sample becomes a pytest test. The judge is a GEval metric (G-Eval: the judge writes evaluation steps from your criteria, then scores). A test fails when the score is below threshold.

# test_capitals.py
import pytest
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams

from my_app import answer   # your system under test: str -> str

correctness = GEval(
    name="Correctness",
    criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
    threshold=0.5,
)

@pytest.mark.parametrize("question,expected", [
    ("What is the capital of France?", "Paris"),
    ("What is the capital of Australia?", "Canberra"),
])
def test_capital(question, expected):
    tc = LLMTestCase(input=question, actual_output=answer(question), expected_output=expected)
    assert_test(tc, [correctness])

# Run:  deepeval test run test_capitals.py

If your team already lives in pytest, this feels native. The trade-off: pytest thinks in pass/fail per test, while evals think in rates across a dataset (Module 04). One flaky judge call fails the build, so thresholds and repeats need care.

4. Braintrust

One call to Eval() with data, task and scores. The result is uploaded as an Experiment that the UI compares with previous experiments row by row.

# capitals_eval.py
from braintrust import Eval
from autoevals import Factuality     # an LLM-as-judge scorer from autoevals

from my_app import answer

def task(input):
    return answer(input)             # the system under test

Eval(
    "Capitals",                      # project name
    data=[
        {"input": "What is the capital of France?", "expected": "Paris"},
        {"input": "What is the capital of Australia?", "expected": "Canberra"},
    ],
    task=task,
    scores=[Factuality],             # judge compares output with expected
)

Langfuse (run_experiment(data, task, evaluators)), LangSmith (evaluate(target, data, evaluators)), MLflow (genai.evaluate(data, predict_fn, scorers)), Weave (Evaluation(dataset, scorers).evaluate(model)) and Opik (evaluate(dataset, task, scoring_metrics)) all have this same three-argument shape. Learn one and you can read the others.

5. OpenAI Evals API (for migration and reading old code)

from openai import OpenAI
client = OpenAI()

# 1) The Eval: what a row looks like + how to grade. No data, no model yet.
eval_obj = client.evals.create(
    name="Capitals",
    data_source_config={
        "type": "custom",
        "item_schema": {
            "type": "object",
            "properties": {"question": {"type": "string"}, "answer": {"type": "string"}},
            "required": ["question", "answer"],
        },
        "include_sample_schema": True,
    },
    testing_criteria=[{
        "type": "score_model",                  # LLM judge that returns a number
        "name": "correct_capital",
        "model": "o4-mini-2025-04-16",
        "input": [
            {"role": "system", "content": "You are an expert grader. Score 1 if the model answer names the reference city as the capital, else 0."},
            {"role": "user", "content": "Reference: {{ item.answer }}. Model answer: {{ sample.output_text }}"},
        ],
        "range": [0, 1],
        "pass_threshold": 0.5,
    }],
)

# 2) The Run: this data + this model + this prompt, graded by the Eval above.
run = client.evals.runs.create(
    eval_obj.id,
    name="gpt-4o baseline",
    data_source={
        "type": "responses",
        "model": "gpt-4o",
        "input_messages": {"type": "template", "template": [
            {"role": "user", "content": "{{ item.question }}"},
        ]},
        "source": {"type": "file_id", "id": "YOUR_FILE_ID"},   # uploaded JSONL of items
    },
)

What the five shapes teach

Rosetta stone: the same concepts, different names

Every tool has the same five or six concepts under different names. This table is worth bookmarking; it lets you read any tool's docs in minutes.

Which tool should I use?

Start from the question you need answered, not from a feature list. The tree below covers most real situations.

The same tree, as rules of thumb

How tools compose

In a mature team the families click together into a loop. Production traces become datasets; datasets feed offline evals; offline evals gate deploys in CI; online judges score live traffic; failing traces become new dataset rows. Each box can be a different tool.

Three concrete, common stacks:

• Lean open-source stack: OpenTelemetry tracing into self-hosted Langfuse or Phoenix; datasets exported from failing traces; promptfoo or DeepEval in GitHub Actions as the gate; Ragas metrics if retrieval matters.
• All-in-one hosted: Braintrust or LangSmith for tracing, datasets, experiments, CI evals and online scoring, with autoevals or openevals judges.
• Research lab style: Inspect tasks with sandboxes for capability and safety evals, Inspect Scout to scan transcripts for anomalies, and Harbor for Terminal-Bench style CLI-agent runs. No production tracing at all, because the "product" is a model.

Open source vs hosted: licenses, lock-in, privacy

Licenses are a spectrum, not a switch

Where your data goes

Eval data is often your most sensitive data: real user conversations, internal documents retrieved by RAG, customer records seen by tools. Before choosing, trace the path each byte takes.

Lock-in: what actually traps you

Switching tools is usually cheap for code and expensive for history. Rank the lock-in risks:

1. Your datasets and labels. Months of curated failure cases and human annotations are the real asset. Make sure you can export them (JSONL or CSV) and keep a copy in your own repo or bucket.
2. Experiment history. Baselines you compare against. If you switch, re-run your current best system on the new tool to create a new baseline; do not compare scores across tools, because graders and prompts differ.
3. Instrumentation. Vendor decorators across your codebase. Prefer OpenTelemetry where offered.
4. Judge prompts. A built-in "Factuality" or "Correctness" judge is someone else's prompt, and it may change between versions. For anything you gate a release on, own the judge prompt and version it (Module 13).
5. Eval code shape. The least sticky: as the Rosetta table shows, the data, task, scorers shape ports in an afternoon.

Check yourself

Sources and further reading

• UK AISI and Meridian Labs, Inspect documentation; Agents; ReAct agent; inspect_ai on GitHub; Meridian Labs, About.
• OpenAI, Evals guide (deprecation notice), Graders, Deprecations, Moving from OpenAI Evals to Promptfoo; openai/evals; openai/simple-evals.
• OpenAI, OpenAI to acquire Promptfoo (2026-03-09); TechCrunch, coverage.
• promptfoo, GitHub, Configuration guide, GitHub Action.
• Braintrust, Eval SDK quickstart; autoevals.
• LangChain, LangSmith evaluation quickstart; langsmith-sdk.
• Langfuse, GitHub, Experiments via SDK.
• Arize, Phoenix on GitHub, Phoenix license, arize-phoenix-evals.
• Confident AI, DeepEval on GitHub.
• Ragas on GitHub; TruLens on GitHub, TruLens quickstart.
• Weights & Biases, Weave on GitHub, Weave evaluations.
• Comet, Opik on GitHub, Opik evaluation docs.
• MLflow, GitHub, GenAI evaluation quickstart.
• EleutherAI, lm-evaluation-harness; Stanford CRFM, HELM; Laude Institute, Harbor; Hugging Face, Lighteval; Groq, OpenBench.
• Pydantic, Pydantic Evals; Google, ADK evaluation, Gen AI evaluation overview, python-aiplatform; Microsoft, Foundry observability and evaluation.
• Anthropic, Demystifying evals for AI agents (2026-01-09); Evaluate prompts in the Console (2024-07-09); Petri; Bloom.
• Patronus AI, Products.
• UK AISI, inspect_evals on GitHub (community collection of Inspect evals).
• NVIDIA, garak: the LLM vulnerability scanner.
• Microsoft, PyRIT on GitHub and PyRIT documentation.
• Giskard, giskard-oss on GitHub and library documentation.
• Galileo, Galileo platform; Galileo introduces evaluation foundation models (Luna); Galileo announces free agent reliability platform.
• Strands Agents, Strands Agents documentation (see its evaluation section for Strands Evals).
• Transluce, Docent.
• Artificial Analysis, Intelligence benchmarking methodology; OpenRouter, Rankings.
• OpenAI, Trace grading, Evaluate agent workflows, Introducing AgentKit (Oct 2025).