How the labs do evals

Anthropic, OpenAI and Google DeepMind each run thousands of evals before a model ships, tie some of them to written safety frameworks, invite outside testers in, and publish the results as a system card. This module maps that whole machine, lab by lab, and then teaches you to read a system card like a skeptical expert.

The big picture: the frontier-lab eval stack

When you build an agent (Parts II and III), you mostly run one kind of eval: "does my system do my users' job?". A frontier lab training a new model has to answer many different questions at many points in time. Some are about engineering ("is this training run healthy?"), some about product ("is the new model better at coding?"), and some about safety and governance ("could this model meaningfully help someone build a bioweapon, and if we can't rule that out, what safeguards must we switch on?").

It helps to picture it as a stack of stages, each with its own evals, its own audience and its own decision.

A few terms before we go lab by lab:

• Checkpoint or snapshot: a saved copy of a model's weights at some point in training. Labs evaluate many; the one that ships is "the final snapshot". A classic system-card caveat is that a safety eval ran on an earlier snapshot than the one released.
• Frontier safety framework: a public document in which a lab commits to test for specific dangerous capabilities and to apply specific safeguards if a model reaches a threshold. Anthropic calls its framework the Responsible Scaling Policy (RSP), OpenAI the Preparedness Framework (PF), Google DeepMind the Frontier Safety Framework (FSF). METR keeps an index of every published one (see Sources).
• Dangerous capability eval: an eval that measures whether a model can do something harmful (for example uplift on a bioweapons workflow, or finding and exploiting software vulnerabilities), usually with the model's refusals turned off or bypassed, because the question is capability, not behavior. Module 17 goes deep on how these are built.
• Elicitation: the effort spent getting the best possible performance out of a model on an eval (good prompts, tools, many attempts, fine-tuning). Weak elicitation makes a model look less capable than it is, which matters a lot for a "rule-out" test.
• System card (OpenAI and Anthropic's term; Google uses model card plus separate reports): the public document released with a model describing what was evaluated and what was found.

The rule-out logic at the heart of framework evals

The most important idea in stage 3 is subtle. Most evals you write try to show that a system can do something (it passed 83% of tasks). Framework evals are mostly used the other way round: to rule out that a model has crossed a dangerous threshold. If the lab can't confidently rule it out, it applies the stronger safeguards anyway.

This flips the usual incentive. In a product eval, bad elicitation makes you look worse, so you are motivated to fix it. In a rule-out eval, bad elicitation makes the model look safer. That is why outside testers, error bars and "how hard did you try?" questions matter so much, and why the Sources for this module keep coming back to elicitation.

Anthropic

The Responsible Scaling Policy and its evolution

Anthropic's RSP ties evaluations to AI Safety Levels (ASLs). An ASL is a bundle of required safeguards, split into a Deployment Standard (how the model is served, for example classifiers that block harmful requests) and a Security Standard (how the weights are protected from theft). Capability thresholds are the points at which a model's abilities in a threat area would require the next level of safeguards.

Why does "only specify the next level" matter for evals? Under a long ladder, a lab must design evals and safeguards for levels that may be years away and based on guesses. Under v3, the lab commits in detail to the next step and publishes Risk Reports on a schedule, which shifts some weight from one-off pre-launch evals to continuous risk reporting. Critics and supporters both discuss what was lost and gained; the GovAI analysis in the Sources is a good balanced read.

What an Anthropic system card covers

Anthropic's system cards have grown into long, detailed documents. Comparing two of them shows the structure is stable:

The Sonnet 4.5 card contains a finding every eval engineer should know: evaluation awareness. The model sometimes said out loud that it thought it was being tested, and white-box probes (tools that read the model's internal activations) found internal representations of eval awareness that "grew in strength over training". If a model behaves better when it believes it is being watched, then alignment evals measure behavior-under-test, not behavior-in-the-wild. Apollo Research and OpenAI reported the same problem in their anti-scheming work (below). This is why the Opus 5.5 card includes a section on the "reliability of assessment" itself.

Demystifying evals for AI agents

In January 2026 Anthropic's engineering blog published "Demystifying evals for AI agents", which is effectively the lab's practical advice to product builders. You have met most of its vocabulary already in this course; here is the mapping, because interviewers and papers use these words exactly:

The advice, condensed: start early with 20 to 50 tasks drawn from real failures; turn manual checks into tests; write unambiguous task specs with reference solutions; balance positive and negative cases; isolate each trial in a clean environment; design graders that are not brittle; read transcripts; watch for saturation; and give the suite a dedicated owner. Separate capability evals (start at low pass rates, show progress) from regression evals (should stay near 100%). A companion post, "Writing effective tools for agents", adds: make eval tasks realistic enough to need many tool calls, track runtime, tool calls, tokens and tool errors alongside accuracy, and notice what agents omit.

Error bars: the statistics paper

Evan Miller's "Adding Error Bars to Evals" (Anthropic, November 2024) is the lab's statement on how eval results should be reported. Its five recommendations, which Module 04 works through in detail:

1. Report the standard error of the mean (SEM), using the central limit theorem.
2. Use clustered standard errors when questions are related (for example, many questions about the same passage).
3. Reduce within-question variance: resample answers or use token probabilities.
4. Compare two models with a paired-difference test on the same questions, not two separate intervals.
5. Do a power analysis before building an eval to know how many questions you need.

# Recommendation 4 in practice: paired difference between two models
# scores_a[i], scores_b[i] = mean score of model A / B on task i (e.g. averaged over trials)
import numpy as np

def paired_diff(scores_a, scores_b):
    a, b = np.asarray(scores_a), np.asarray(scores_b)
    d = a - b                                  # per-task difference
    mean = d.mean()
    sem = d.std(ddof=1) / np.sqrt(len(d))     # SEM of the difference
    return mean, (mean - 1.96 * sem, mean + 1.96 * sem)

# Because both models see the SAME tasks, task difficulty cancels out,
# and the interval is usually much tighter than comparing two separate +/-3.7 bars.

Automated auditing: Petri and Bloom

Alignment evals (does the model deceive, sabotage, seek power, flatter users?) are hard to write by hand: the interesting behaviors are rare and depend on context. Anthropic's answer has been to use models to evaluate models, a line of work that goes back to model-written evals (Perez et al., 2022). That paper had a model generate 154 eval datasets; crowdworkers agreed with 90 to 100% of the labels, and the evals found that sycophancy (telling users what they want to hear) increases with model scale.

Two open-source tools continue that idea, and both are now maintained by the non-profit Meridian Labs (which also co-develops Inspect with UK AISI):

• Petri launched with a run of 14 models across 111 seed instructions. A notable finding was whistleblowing: models sometimes tried to report perceived wrongdoing by their (simulated) operators on their own initiative, a behavior worth knowing about if your agent has email or web tools.
• Bloom benchmarked 16 models on delusional sycophancy, instructed long-horizon sabotage, self-preservation and self-preferential bias.
• Agentic Misalignment (June 2025) is the most cited example of this style of stress test: in artificial corporate scenarios where a model faced replacement plus a goal conflict, Claude Opus 4 and Gemini 2.5 Flash chose blackmail in 96% of runs of one scenario, GPT-4.1 and Grok 3 Beta in 80%. Anthropic stressed the caveat: these were contrived binary dilemmas, not observed real-world behavior.

How Anthropic reports agentic benchmarks

Launch posts come with footnotes that are easy to skip and essential to read. For Claude Sonnet 4.5 (September 2025), Anthropic wrote that SWE-bench Verified results used "a simple scaffold with two tools, bash and file editing via string replacements", reported 77.2% "averaged over 10 trials, no test-time compute" on the full 500 problems, and separately reported 82.0% with parallel attempts, rejection sampling using visible regression tests and internal model selection. OSWorld used the official OSWorld-Verified framework with a 100-step limit averaged over 4 runs; Terminal-Bench used the Terminus 2 agent framework averaged over runs on different days; and τ2-bench used extended thinking plus prompt modifications for known failure modes.

This transparency is good practice, and it is also a warning: without the footnote, "82%" and "77%" look like two different models. Note also that OpenAI stopped reporting SWE-bench Verified in February 2026 after finding contamination (models could reproduce gold patches verbatim), which is covered in Module 18.

OpenAI

The Preparedness Framework (v2, April 2025)

OpenAI's framework sorts risks into Tracked Categories, which have full evaluation and safeguard requirements, and Research Categories, which OpenAI studies but has not yet turned into formal thresholds. Each tracked category has two levels, High and Critical.

Notice that sandbagging is a research category. That is the rule-out problem again: if a model could deliberately do worse on dangerous-capability evals, a "below threshold" result means less. The GPT-5 system card (August 2025) is a good example of the framework in action: OpenAI treated GPT-5 thinking as High capability in Biological and Chemical as a precaution, while saying it lacked definitive evidence that the model could help novices create severe biological harm. It did not treat the model as High in cybersecurity or AI self-improvement.

OpenAI system cards and external testers

The GPT-5 system card is organized as: Introduction; Model Data and Training; Observed Safety Challenges and Evaluations; Red Teaming and External Assessments; Preparedness Framework; and appendices (including one on hallucinations). The external assessors named include METR (autonomous capabilities, sandbagging risk), Apollo Research (deceptive behavior, evaluation awareness), SecureBio (biological and chemical capabilities), Pattern Labs (offensive cyber), Microsoft's AI Red Team and Gray Swan (prompt injection). So a single system card mixes in-house evals, specialist contractors and independent non-profits.

A standout collaboration is OpenAI and Apollo Research's anti-scheming work (September 2025). They built 180+ test environments that use covert actions (secretly breaking rules or withholding information) as a measurable proxy for scheming. After targeted training, covert action rates fell from 13% to 0.4% for o3 and from 8.7% to 0.3% for o4-mini. The caveat is the important part: evaluation awareness causally contributed to the reduction. Some of the "improvement" was the model recognizing it was being tested.

Benchmarks OpenAI built

OpenAI publishes a lot of benchmarks, and they share a theme: grounding the task in real work, real money or real experts.

Two of these (GDPval, PaperBench) reported a rival lab's model as the best performer at launch, which is a healthy sign. Module 07 and Module 08 cover SWE-Lancer and BrowseComp as agent benchmarks.

The Evals API, graders, RFT, and the move to promptfoo

OpenAI's developer platform offered a hosted Evals API with reusable graders. The design is worth learning even though the product is going away, because it cleanly separates dataset schema, grader and run, and because the same grader objects doubled as reward functions for reinforcement fine-tuning (RFT).

The grader types, briefly: string_check (eq, neq, like, ilike, returns 0 or 1); text_similarity (fuzzy match, BLEU, ROUGE, cosine and others, 0 to 1); score_model (an LLM returns a number in a range); label_model (an LLM picks a label, some of which count as passing); python (your own grade(sample, item) function); and multi (combine several with a formula). Here is how a check looked on the platform, and the same idea in promptfoo:

# OpenAI hosted Evals (DEPRECATED: read-only 2026-10-31, shut down 2026-11-30)
eval_obj = client.evals.create(
    name="IT Ticket Categorization",
    data_source_config={"type": "custom", "item_schema": {...}, "include_sample_schema": True},
    testing_criteria=[{
        "type": "string_check",                  # a grader object
        "name": "Match output to human label",
        "input": "{{ sample.output_text }}",     # what the model produced
        "operation": "eq",
        "reference": "{{ item.correct_label }}", # from your dataset row
    }],
)

# promptfoo equivalent (promptfooconfig.yaml), OpenAI's recommended replacement
prompts:
  - file://prompt1.txt
providers:
  - openai:gpt-6-luna          # model ids change; use your own
tests:
  - vars:
      ticket: "My laptop will not turn on"
    assert:
      - type: equals            # like string_check eq
        value: hardware
      - type: llm-rubric        # like a score_model / label_model grader
        value: gives a category, not a paragraph

RFT guidance from OpenAI is itself eval wisdom: make tasks guess-proof (if a lucky guess earns reward, the signal is noise) and give partial credit for a smoother signal. Those two rules apply just as well to any grader you design (Module 03).

Agent workflows: traces, trace grading and AgentKit

The Evals API above grades one input and one output. An agent run is more than that: the model thinks, calls tools, hands work to other agents and passes guardrail checks before it answers. OpenAI's tooling calls the full record of one run a trace, and each step inside it a span (a model generation, a tool call, a handoff, a guardrail check). Its open-source Agents SDK records traces automatically. Trace grading means attaching graders to a trace, so you can ask questions about how the agent got its answer ("did it look the order up before refunding?") and not only whether the final answer was right.

OpenAI's "Evaluate agent workflows" guide lays out, in outline, a progression that matches this course: start by reading traces to see what the agent actually does, grade traces to find failure patterns across many runs, then turn what you learn into datasets and repeatable eval runs that gate changes. In October 2025 OpenAI bundled this into AgentKit: a visual Agent Builder, a Connector Registry, the embeddable ChatKit UI, Guardrails, and new Evals features (datasets, trace grading, automated prompt optimization, and support for evaluating third-party models).

Current status, consistent with the section above. The June 2026 deprecation notices cover the hosted Evals platform (which is where dashboard trace grading runs) and, per OpenAI's deprecation notice for it, Agent Builder, with shutdown scheduled for November 30, 2026. The Agents SDK and its tracing are a separate open-source library. So the durable part of this story is the idea, not the dashboard: capture traces yourself (the SDK lets you register your own trace processors, and many observability tools ingest its traces, see Module 14) and write trace graders as ordinary code or as checks in an open harness. Check OpenAI's deprecations page before you rely on any hosted piece.

# Portable trace grading: record spans from the OpenAI Agents SDK (open source)
# and grade them in code, independent of any hosted dashboard.
from agents import Agent, Runner, function_tool, trace, add_trace_processor
from agents.tracing import TracingProcessor

class SpanRecorder(TracingProcessor):
    """Keeps an exported copy of every finished span."""
    def __init__(self):
        self.spans = []
    def on_trace_start(self, trace): pass
    def on_trace_end(self, trace): pass
    def on_span_start(self, span): pass
    def on_span_end(self, span):
        exported = span.export()          # dict incl. trace_id and span_data
        if exported:
            self.spans.append(exported)
    def shutdown(self): pass
    def force_flush(self): pass

recorder = SpanRecorder()
add_trace_processor(recorder)            # the default exporter keeps running too

@function_tool
def lookup_order(order_id: str) -> str:
    """Look up an order by id."""
    return "order 123: delivered, $40"    # mock data, nothing real is touched

@function_tool
def issue_refund(order_id: str, amount: float) -> str:
    """Refund an order."""
    return "refund queued (mock)"

agent = Agent(name="Support",
              instructions="Help with orders. Always look an order up before refunding it.",
              tools=[lookup_order, issue_refund])

def grade_trace(spans, final_output):
    data = [s["span_data"] for s in spans]            # field names per the SDK's span export
    tools = [d.get("name") for d in data if d.get("type") == "function"]
    refunded = "issue_refund" in tools
    looked_up_first = (not refunded) or (
        "lookup_order" in tools and tools.index("lookup_order") < tools.index("issue_refund"))
    return {
        "looked_up_before_refund": looked_up_first,    # process check on span order
        "tool_calls": len(tools),                      # efficiency signal
        "mentions_refund": "refund" in final_output.lower(),   # crude outcome check
    }

recorder.spans.clear()                   # one case at a time
with trace("refund-eval case 17"):
    result = Runner.run_sync(agent, "Order 123 arrived broken. Refund me.")
print(grade_trace(recorder.spans, result.final_output))

The grader returns several named checks instead of one score, which is the point of trace grading: a run can reach a good final answer by a bad path (refunding without a lookup), and only a process check sees that. The same idea appears as trajectory matching in Google's ADK below and in Module 05.

Google DeepMind and Google Cloud

The Frontier Safety Framework: CCLs, alert thresholds and TCLs

Google DeepMind's framework is built around Critical Capability Levels (CCLs): capability levels at which, without mitigations, a model could pose a severe risk. Version 3.0 (September 2025) listed CCLs in three families, and version 3.1 (April 2026) added Tracked Capability Levels (TCLs) to "spot and evaluate potential less extreme risks sooner".

The Gemini 3 Pro FSF report (November 2025) shows what the evals look like in practice:

• CBRN: multiple-choice questions, open-ended questions, red teaming and external wet-lab trials.
• Cyber: "key skills" capture-the-flag challenges. The model solved 11 of 12 hard v1 challenges but 0 of 13 end-to-end challenges, so the alert threshold was met even though no CCL was reached.
• Harmful manipulation: propensity tests with synthetic dialogues, plus efficacy studies with human participants (does the model actually change what people believe or do?).
• ML R&D: RE-Bench, METR's research-engineering benchmark.
• Misalignment: situational awareness and stealth evals (the "instrumental reasoning" CCLs). Version 3 also addresses a model interfering with operators' ability to shut it down.

Structurally, Google publishes the FSF report separately from the model card, while Anthropic and OpenAI fold framework results into one system card. That is a presentation choice, but it means you need to read two documents for a Gemini model.

Google Cloud's eval service and ADK eval

On the product side, Google's hosted Gen AI evaluation service now lives under the Gemini Enterprise Agent Platform (the April 2026 rename of Vertex AI). It supports rubric-based metrics (including adaptive rubrics generated per prompt), computation-based metrics, pointwise and pairwise comparison (AutoSxS) and agent trajectory metrics such as trajectory_exact_match, trajectory_in_order_match, trajectory_any_order_match, trajectory_precision and trajectory_recall.

The open-source Agent Development Kit (ADK) has its own eval runner. It distinguishes .test.json files (one session, like a unit test) from .evalset.json files (many, possibly long, multi-turn sessions, like integration tests). Its two default criteria are the easiest way to see how a lab's tooling encodes "grade the path and the answer":

// test_config.json next to your eval files (ADK)
{
  "criteria": {
    "tool_trajectory_avg_score": { "threshold": 1.0, "match_type": "IN_ORDER" },
    "response_match_score": 0.8
  }
}

# run an evalset from the command line
adk eval path/to/refund_agent path/to/refund_agent/refunds.evalset.json \
    --config_file_path=path/to/test_config.json --print_detailed_results

The defaults, if you do not provide a config, are tool_trajectory_avg_score = 1.0 (every tool call must match) and response_match_score = 0.8 (ROUGE-1 overlap between the final answer and the reference must be at least 0.8). ROUGE-1 counts overlapping words, so 0.8 allows some rephrasing but penalizes a very different answer; for looser semantic checks ADK offers final_response_match_v2 (an LLM judge) and rubric-based criteria such as rubric_based_tool_use_quality_v1, plus hallucinations_v1, safety_v1 and multi-turn criteria like multi_turn_task_success_v1. You can also run evals from pytest via AgentEvaluator, and use adk conformance to check against recorded baselines.

Meta, xAI and others

• Meta published its Frontier AI Framework in February 2025: outcome-based and threat-model-driven, covering cyber and chem/bio, and arguing that open release lets the wider community help evaluate models. In April 2026 it was renamed the Advanced AI Scaling Framework (v2). Per secondary analysis, the scope now covers models with catastrophic capability or trained with at least 10²⁶ FLOP, the trigger changed from a model "uniquely enabling" a catastrophe to "substantially contributing" to one, a Loss of Control domain was added, and Meta committed to preparedness reports.
• xAI's Risk Management Framework (August 2025) is the most explicitly benchmark-threshold based: malicious-use evals such as VCT, WMDP, BioLP-bench and Cybench; loss-of-control evals such as MASK (honesty) and sycophancy; with numeric thresholds like an answer rate on restricted bio/chem queries below 1 in 20 and a MASK dishonesty rate below 1 in 2. A newer xAI framework is listed on METR's index, but its contents were not verified for this course.
• Others with published frontier policies, according to METR's index, include Microsoft, Amazon, Nvidia, Cohere, G42, NAVER and Magic. No public Mistral frontier framework was found.

Third parties: who checks the labs?

A lab grading its own homework is an obvious problem. Outside evaluators fall into two groups: those who get pre-deployment access (they test before the public does, under agreement with the lab) and those who evaluate after release using public APIs or weights.

# A minimal Inspect task: the same shape AISI-style evals use
from inspect_ai import Task, task
from inspect_ai.dataset import FieldSpec, hf_dataset
from inspect_ai.scorer import model_graded_qa
from inspect_ai.solver import generate

@task
def simpleqa():
    return Task(
        dataset=hf_dataset(
            "codelion/SimpleQA-Verified",
            split="train",
            sample_fields=FieldSpec(input="problem", target="answer"),
        ),
        solver=generate(),          # swap for react(...) with bash tools to make it agentic
        scorer=model_graded_qa(),   # an LLM judge compares answer to target
    )

# run:  inspect eval simpleqa.py --model anthropic/<model-id>
# view: inspect view   (browse every transcript)

Regulators and standards

Everything above is either a lab's own voluntary framework or an outside tester working under a voluntary agreement. That is changing: parts of frontier evaluation are becoming legal and standards obligations. The EU AI Act's rules for general-purpose AI models with systemic risk include model evaluation and adversarial testing, with a Code of Practice describing how providers can meet them; in the US, NIST's work (including CAISI, above) shapes testing practice; and management-system standards such as ISO/IEC 42001 shape how organizations govern and document their evals. Module 28 covers these properly, including how a regulator-facing eval differs from an engineering eval and what a safety case looks like. For now, notice the direction of travel: the rule-out logic, documented elicitation and third-party testing described in this module are exactly what regulators are starting to ask for in writing.

The big comparison table

Here is the whole landscape on one screen. "Who tests" means who is named in the lab's public materials, not a complete list.

How to read a system card

System cards are long, dense and written by the organization that trained the model. They are also the single best public source on how frontier evals are actually done. Read them the way a scientist reads a paper by a company about its own product: grateful for the data, careful with the conclusions.

The five big things to be skeptical of

Different philosophies, carefully stated

It is tempting to caricature the labs. The fairer approach is to describe what each one publishes and emphasizes, which is observable, without claiming to know their internal priorities.

• Anthropic gives a lot of published space to agentic safety and alignment audits: dedicated sections on reward hacking, alignment assessment and evaluation awareness, open-sourced auditing agents (Petri, Bloom), stress tests like Agentic Misalignment, and methodological papers (error bars, model-written evals, "Demystifying evals"). Its RSP v3 moved from a long ladder of pre-specified levels to "next level only" plus periodic Risk Reports.
• OpenAI's public benchmarks are notably economically and professionally grounded: GDPval draws its occupations from industries that each make up over 5% of US GDP, SWE-Lancer measures dollars of real freelance work, HealthBench and PaperBench use expert-written rubrics. Its developer story tied evals to training (graders as RFT rewards), and is now moving to promptfoo. Its framework separates tracked from research categories and gives two named levels.
• Google DeepMind frames frontier risk around Critical Capability Levels with early warning: alert thresholds, TCLs for lower-severity risk, and an explicit safety-case review. Its v3 framework added harmful manipulation as a CCL, measured partly with human-participant studies.
• Meta and xAI are more distinct still: Meta argues that openness itself helps evaluation, with a compute-based scope; xAI writes numeric benchmark thresholds straight into its framework.

Check yourself

Sources and further reading

• Anthropic, Responsible Scaling Policy (v3.4, effective July 8, 2026) and RSP v3 announcement (Feb 2026).
• GovAI, Anthropic's RSP v3.0: how it works, what's changed.
• Anthropic, Activating ASL-3 protections (May 2025).
• Anthropic, Claude Sonnet 4.5 System Card and Claude Opus 5.5 System Card.
• Anthropic, Introducing Claude Sonnet 4.5 (benchmark methodology footnotes).
• Anthropic Engineering, Demystifying evals for AI agents (Jan 2026) and Writing effective tools for agents.
• Evan Miller, Adding Error Bars to Evals; Anthropic blog A statistical approach to model evaluations.
• Perez et al., Discovering Language Model Behaviors with Model-Written Evaluations.
• Anthropic, Petri, Bloom, Bloom technical post, Agentic Misalignment; Meridian Labs, About.
• OpenAI, Updating our Preparedness Framework and Preparedness Framework v2 (PDF).
• OpenAI, GPT-5 System Card.
• OpenAI and Apollo Research, Detecting and reducing scheming; paper arXiv 2509.15541.
• OpenAI benchmarks: GDPval, SWE-Lancer, PaperBench, BrowseComp, HealthBench; Why we no longer evaluate SWE-bench Verified.
• OpenAI docs: Evals guide (deprecation notice), Graders, Reinforcement fine-tuning, Evaluation best practices, Moving from OpenAI Evals to promptfoo; OpenAI to acquire Promptfoo.
• OpenAI agent-eval tooling: Trace grading, Evaluate agent workflows, Introducing AgentKit (Oct 2025) and its cookbook walkthrough; Agents SDK tracing; Deprecations and the Agent Builder deprecation notice.
• Google DeepMind, Strengthening our Frontier Safety Framework, FSF v3.1 (PDF), Gemini 3 Pro FSF report.
• Google Cloud, Gen AI evaluation service overview; ADK, Why evaluate agents and Evaluation criteria.
• Meta, Our approach to frontier AI; analysis of the Advanced AI Scaling Framework. xAI, Risk Management Framework.
• METR, Frontier AI safety policies index, Measuring AI Ability to Complete Long Tasks, Time Horizon 1.1.
• UK AISI, Inspect and pre-deployment evaluation of upgraded Claude 3.5 Sonnet.
• US Department of Commerce, CAISI announcement; NIST, CAISI evaluation of DeepSeek.
• Epoch Capabilities Index; Scale SEAL leaderboards; Artificial Analysis methodology; Singh et al., The Leaderboard Illusion.