Vision, image and video

Pixels make evaluation harder in two opposite directions. When a model reads images, you must prove it actually looked. When a model makes images, there is no single right answer to compare against. This module covers both, then puts them together in agents that see, create and revise.

Two directions, one problem

A vision-language model (VLM) is a model that accepts images (and sometimes video frames) as input alongside text. Claude, GPT and Gemini models are all VLMs today. A generative image model goes the other way: text (and optionally an image) in, pixels out. Diffusion models such as Stable Diffusion, and video generators, are in this family. Agents increasingly use both: a design agent looks at a screenshot, calls an image generator, looks at the result, and edits again.

Part A: evaluating image and video understanding

Understanding evals look like ordinary text evals (Module 02) with an image attached. The format that dominates is VQA, "visual question answering": an image, a question about it, and a reference answer. The differences hide in two places: what the question actually requires, and how the answer is graded.

The main benchmarks

No scores are quoted here on purpose. Frontier scores on these benchmarks move every few months, and several (MMMU, MathVista, ChartQA, DocVQA) are near their ceilings for top models. Check each benchmark's official leaderboard, with a date, when you need a number.

How understanding answers are graded

You met these grader types in Module 03. Here is how they apply to vision.

The big pitfall: did the model actually look?

The most important idea in vision understanding evals is that a question can be answerable without the image. That happens for two reasons: the question text gives the answer away ("What color is the sky in this clear daytime photo?"), or the model has memorized the item from training data. Either way the benchmark stops measuring vision.

The MMStar paper (Chen et al., 2024) measured this directly. Its headline example: Gemini Pro scored 42.9% on MMMU with no image at all, and across six popular benchmarks, models running without images beat the random-choice baseline by over 24% on average. MMMU-Pro was built partly as a response: it filters out questions that text-only models can solve.

Here is the whole idea as code. It works with any VLM client; the function ask is whatever you already use to call your model.

# Blind-baseline audit for a multiple-choice vision benchmark.
# ask(question, options, image_or_None) -> a letter like "B"
import random
from collections import Counter

def audit(items, ask, n_blind_samples=3):
    with_img = blind = 0
    leaky = []
    for it in items:
        if ask(it["question"], it["options"], it["image"]) == it["answer"]:
            with_img += 1
        # Ask several times without the image; a single lucky guess is not a leak.
        votes = Counter(ask(it["question"], it["options"], None)
                        for _ in range(n_blind_samples))
        top, count = votes.most_common(1)[0]
        if top == it["answer"]:
            blind += 1
            if count == n_blind_samples:          # consistently right without looking
                leaky.append(it["id"])
    n = len(items)
    chance = sum(1 / len(it["options"]) for it in items) / n
    print(f"with image {with_img/n:.1%}  blind {blind/n:.1%}  chance {chance:.1%}")
    print(f"visual gain {(with_img - blind)/n:.1%}; {len(leaky)} items look blind-answerable")
    return leaky

Resolution, cropping and the invisible detail

Every VLM resizes images before it reads them. Claude's documentation, for example, says images above a model's maximum are downscaled while preserving aspect ratio: 1568 px on the long edge for most models, 2576 px for the newer high-resolution tier (Anthropic vision docs, fetched September 2026). Other providers tile the image into crops, or let you choose a "detail" level. Image cost scales with pixels too: Claude counts roughly one visual token per 28 × 28 pixel patch, so a 1920 × 1080 screenshot is shrunk to 1456 × 819 (about 1,560 tokens) on the standard tier but sent whole (about 2,700 tokens) on the high-resolution tier. This matters for evals in two ways.

• Preprocessing is part of the harness. Two teams reporting DocVQA for the same model can differ because one tiles pages and the other sends a single downscaled image. When you compare numbers, compare preprocessing too, just as Module 07 insists on comparing scaffolds for coding agents.
• Failures that look like reasoning are often perception. In error analysis (Module 12), add a label for "could not see it" separate from "saw it, reasoned wrong". The fix for the first is cropping, zoom tools or higher resolution; for the second it is a better model or prompt.
• Agents can zoom. A computer-use agent (Module 08) that can take a closer screenshot or crop is a different system from one that cannot. Test both if your product allows it.

OCR shortcuts and other ways to score without understanding

OCR (optical character recognition) means reading text out of an image. Many "vision" tasks can be solved by OCR alone: read every word on the chart, then do text reasoning. That is legitimate for DocVQA (reading is the point) but misleading for a benchmark that claims to test spatial or visual reasoning. Warning signs:

• A pipeline of "OCR tool + text-only LLM" scores close to the VLM. The benchmark mostly tests reading.
• Chart questions whose answer is printed as a data label on the chart. The model copies a number instead of comparing bar heights. ChartQA's human-written questions were collected partly to require visual and logical reasoning rather than lookup.
• Video questions answerable from subtitles. That is why Video-MME reports results with and without subtitles. MVBench's known issue is the video version of the blind question: some items are answerable from a single frame, so they do not test temporal understanding.

Long video and the visual needle in a haystack

Long-context text models are tested with needle in a haystack: hide one fact in a very long document and ask for it. The video version inserts a "needle" (an out-of-place image or short clip with a question about it) somewhere inside a long video. Zhang et al.'s V-NIAH (June 2024, from the LongVA paper) is the standard example: it measures whether a model can find and use one piece of visual information buried among thousands of frames' worth of visual tokens.

Part B: evaluating image and video generation

Now flip direction. Given "a red cube on top of a blue sphere", there are infinitely many correct images. So instead of one grader, generation evals combine several questions:

1. Quality / realism: does it look like a real, well-made image? (FID, aesthetic predictors, human ratings)
2. Alignment / faithfulness: does it show what the prompt asked for? (CLIPScore, GenEval, T2I-CompBench, TIFA, DSG, VQAScore)
3. Preference: which of two images do people prefer? (arenas, reward models)
4. Temporal behavior (video only): is motion smooth, does the subject stay the same? (VBench)
5. Safety and provenance: is it allowed, and is it labeled as AI-made? (classifiers, watermarks, C2PA)

FID: comparing two clouds of images

FID, the Fréchet Inception Distance (Heusel et al., 2017), asks: "Do the generated images, as a population, look like real images, as a population?" It never looks at a single image in isolation, and it never looks at the prompt.

The recipe: pass a few thousand real images and a few thousand generated images through a fixed pretrained network (Inception-v3), take a 2,048-number feature vector from near the end of the network for each image, and fit a Gaussian (a mean vector and a covariance matrix) to each set. FID is the distance between the two Gaussians. Lower is better; 0 means the two feature distributions are identical.

Limits of FID

• No prompt. A text-to-image model that ignores prompts but draws lovely photos can get a good FID.
• Sample-size bias. FID estimated from fewer images is biased upward, so scores computed with different sample counts are not comparable. Report the count (commonly 10K to 50K images) and the reference set (for text-to-image, often MS-COCO captions and images).
• Implementation details move it. Image resizing method, JPEG compression and the exact Inception weights all change the number. Use one library (for example the clean-fid package) for every model you compare.
• Feature bias. Inception-v3 was trained on ImageNet classification. It may not care about text rendering, hands or artistic style. Jayasumana et al. ("Rethinking FID", CVPR 2024) argue FID can contradict human judgment and propose CMMD, which uses CLIP features and a distance that does not assume Gaussians.

CLIPScore: does the image match the text?

CLIP (OpenAI, 2021) is a pair of encoders trained so that an image and its caption land near each other in a shared embedding space. CLIPScore (Hessel et al., 2021) uses this directly: embed the image, embed the prompt, take the cosine similarity, and rescale. The paper defines CLIP-S = w × max(cos(image, text), 0) with w = 2.5. Many libraries instead report 100 × cosine. Both are fine as long as you use one convention throughout.

Here is CLIPScore with the open_clip library (API as in the project's README; install with pip install open_clip_torch).

import torch
import open_clip
from PIL import Image

model, _, preprocess = open_clip.create_model_and_transforms(
    "ViT-B-32", pretrained="laion2b_s34b_b79k")
model.eval()
tokenizer = open_clip.get_tokenizer("ViT-B-32")

def clip_score(image_path: str, prompt: str, w: float = 2.5) -> float:
    image = preprocess(Image.open(image_path).convert("RGB")).unsqueeze(0)
    text = tokenizer([prompt])
    with torch.no_grad():
        img = model.encode_image(image)
        txt = model.encode_text(text)
    img = img / img.norm(dim=-1, keepdim=True)   # unit vectors, so dot = cosine
    txt = txt / txt.norm(dim=-1, keepdim=True)
    cos = (img @ txt.T).item()
    return w * max(cos, 0.0)                     # Hessel et al. 2021 definition

# Sanity check the metric before trusting it: score the SAME image against
# the correct prompt and a "swapped" prompt. If the gap is tiny, CLIPScore
# cannot see the thing you care about.
print(clip_score("gen_001.png", "a red cube on top of a blue sphere"))
print(clip_score("gen_001.png", "a blue cube on top of a red sphere"))

Compositional checks: GenEval and T2I-CompBench

Because CLIPScore misses composition, researchers built evals that check specific, verifiable facts about the image. The trick is to turn "is this image faithful?" into several small questions that a specialist vision model can answer reliably.

GenEval (Ghosh, Hajishirzi and Schmidt, NeurIPS 2023 Datasets and Benchmarks) has 553 templated prompts covering single objects, two objects, counting, colors, position and attribute binding. It runs an object detector (Mask2Former) on the generated image, then applies simple rules: are the right objects present, is the count right, is object A left of object B (by comparing box centers), and is each object's color right (by classifying the cropped object with CLIP).

You can build the same kind of check for your own product with an off-the-shelf detector. This sketch uses the Hugging Face transformers object-detection pipeline.

from transformers import pipeline
from PIL import Image

detector = pipeline("object-detection", model="facebook/detr-resnet-50")

def check_count_and_position(image_path, obj, n, left_of=None, thresh=0.7):
    """GenEval-like rule check: exactly n of obj, and (optionally) all left of another."""
    dets = [d for d in detector(Image.open(image_path).convert("RGB"))
            if d["score"] >= thresh]                 # drop low-confidence boxes
    objs = [d for d in dets if d["label"] == obj]
    result = {"count_ok": len(objs) == n, "found": len(objs)}
    if left_of:
        anchors = [d for d in dets if d["label"] == left_of]
        if not anchors:
            result["position_ok"] = False
        else:
            cx = lambda b: (b["xmin"] + b["xmax"]) / 2   # box centre x
            a = cx(anchors[0]["box"])
            result["position_ok"] = all(cx(o["box"]) < a for o in objs)
    result["pass"] = all(v for k, v in result.items() if k.endswith("_ok"))
    return result

# Prompt was generated from a template, so we know what to expect:
# "a photo of three apples to the left of a bowl"
print(check_count_and_position("gen_042.png", "apple", 3, left_of="bowl"))

Two things to calibrate before trusting it. First, the confidence threshold: too high and real objects are missed, too low and hallucinated boxes appear. Hand-label 50 images and pick the threshold that best agrees with you. Second, the label vocabulary: this detector knows the 80 COCO categories, so "croissant" is not checkable with it.

T2I-CompBench (Huang et al., NeurIPS 2023) scales the idea to 6,000 compositional prompts in three groups: attribute binding (color, shape, texture), object relationships (spatial and non-spatial) and complex compositions. It uses a different specialist per group: BLIP-VQA for attributes, the UniDet detector for spatial relations, CLIPScore for non-spatial relations, and a combined "3-in-1" metric for complex prompts. T2I-CompBench++ (TPAMI 2025) adds numeracy and 3-D relations.

VQA-based alignment: TIFA, DSG and VQAScore

Detectors only know object categories. The next step replaces them with a general VLM that can answer any question about the image. This family goes: prompt → questions → answers from the image → score.

• TIFA (Hu et al., ICCV 2023): a language model generates question-answer pairs from the prompt, and a VQA model answers them on the generated image. TIFA v1.0 has 4K prompts and 25K questions across 12 categories. Its authors found models struggled with counting, spatial relations and composing multiple objects, while doing well on color and material.
• DSG, Davidsonian Scene Graph (Cho et al., Google, ICLR 2024): generates atomic, unique questions organized in a dependency graph, which avoids duplicate questions and contradictory answers (a VQA model saying "no dog" but "yes, the dog is black"). DSG-1k has 1,060 prompts.
• VQAScore (Lin et al., CMU and Meta, 2024): skips question generation entirely. It asks one question, "Does this figure show '{prompt}'?", and uses the VQA model's probability of answering "Yes" as the score. The authors also released GenAI-Bench: 1,600 compositional prompts with over 15,000 human ratings, used to check which metric agrees best with people.

Human preference and arenas

Alignment metrics check that the prompt was followed. They do not tell you whether the image is good: attractive, coherent, free of odd hands and melted text. For that, the field uses people.

Arenas show a voter two anonymous images for the same prompt and ask which is better. LMArena runs text-to-image and image-edit arenas, and Artificial Analysis runs text-to-image and video arenas. Votes are aggregated into ratings using Elo or, more commonly now, a Bradley-Terry model, which assumes the probability that model A beats model B is determined by the difference of their hidden strengths, and fits those strengths to all the votes at once (see Module 04 and Module 13).

Arenas have the same problems as text arenas: voters' taste for vivid, saturated images, prompts skewed toward what arena visitors happen to type, and the "Leaderboard Illusion" critique (Singh et al., 2025) of labs privately testing many variants and publishing the best. Use them as a signal about general appeal, not as proof your use case is covered.

When you cannot afford humans for every comparison, learned preference models approximate them: HPSv2, ImageReward and PickScore are models trained on large sets of human choices to predict which image people prefer. They are useful for ranking many candidates quickly, but they are also a common training reward, and a generator optimized against one will learn its quirks. Do not evaluate with the same reward model you trained on (that is Goodhart's law: once a measure becomes a target, it stops being a good measure).

Video generation: VBench

Video adds a time axis, so new failures appear: a character's face changes between frames, the background flickers, objects teleport, or nothing moves at all. VBench (Huang et al., CVPR 2024) splits "video quality" into 16 separate dimensions, each with its own prompt set and an automatic evaluator chosen for that dimension, validated against human preference.

Follow-ups extend it: VBench++ adds image-to-video and trustworthiness dimensions, and VBench-2.0 (2025) targets "intrinsic faithfulness" such as physics, commonsense and human anatomy, which the original automatic metrics could not judge. Known issues: top models cluster tightly on many dimensions, and some automatic dimensions disagree with what viewers perceive as quality. As with images, arenas fill that gap.

Safety and provenance for generated media

Generation evals also have to answer "should this image exist?". Module 17 covers safety evals in general; the image-specific parts are:

C2PA (Coalition for Content Provenance and Authenticity) is an open standard for Content Credentials: signed metadata attached to a file that records its origin and edit history, described by C2PA as "a nutrition label for digital content". Metadata can be stripped, so providers often combine it with invisible watermarks. An eval here is a robustness test: apply common transformations and measure how often the credential or watermark is still detected.

Part C: multimodal agents that see and make

A multimodal agent uses vision and generation as steps in a loop. Examples: a design agent that produces a social-media banner from a brief, looks at its draft, and fixes the logo placement; an image-editing agent that takes "remove the car and make it evening" and applies several tool calls; a slide agent that renders a deck and checks each slide for overflow. Computer-use agents (Module 08) are the closest cousin: they also see screenshots and act, and "did the rendered page look right?" is a shared question.

Layer 1: code checks first

Many requirements are objective and cheap. Is the image 1200 × 628? Is it a PNG under 1 MB? Does OCR find the exact headline "Summer Sale, 30% off" with no misspellings (text rendering is still a common weak spot of image generators)? Is the brand hex color present in the palette? For an edit ("remove the car"), are pixels outside the edit region unchanged? Measure that with a pixel difference or a perceptual similarity metric such as SSIM or LPIPS on the masked region. Code checks are deterministic, so they belong in CI (Module 15).

Layer 2: a VLM judge with a rubric

For everything subjective, use a VLM as the judge: give it the brief, the image and a rubric with separate criteria, and ask for a short reason before each score. Separate criteria stop one strong impression (a gorgeous image) from hiding a failure (the wrong product in it).

Here is a judge call using the Anthropic Python SDK with a base64 image content block (structure from Anthropic's vision documentation; the model id is the one used in the current docs, so swap in whichever model you have chosen as your judge).

import base64, json, pathlib
import anthropic

client = anthropic.Anthropic()   # reads ANTHROPIC_API_KEY from the environment

RUBRIC = """Score each criterion 1-5 using these anchors.
gate: fail if unsafe content OR the headline text is not exactly as in the brief.
adherence: 1 wrong subject, 3 subject ok but a required detail missing, 5 all present.
layout: 1 text over faces or cropped, 3 cramped but legible, 5 clear hierarchy.
brand: 1 wrong logo or colours, 3 colours off, 5 matches the guide.
quality: 1 obvious artefacts (extra fingers, melted text), 3 minor, 5 clean.
Write a one-sentence reason BEFORE each score.
Return only JSON: {"gate": {"reason": str, "pass": bool},
 "adherence": {"reason": str, "score": int}, "layout": {...},
 "brand": {...}, "quality": {...}}"""

def judge(image_path: str, brief: str) -> dict:
    data = base64.standard_b64encode(pathlib.Path(image_path).read_bytes()).decode()
    msg = client.messages.create(
        model="claude-opus-5-5",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": [
                {"type": "image",
                 "source": {"type": "base64", "media_type": "image/png", "data": data}},
                {"type": "text",
                 "text": f"<brief>\n{brief}\n</brief>\n\n<rubric>\n{RUBRIC}\n</rubric>"},
            ],
        }],
    )
    text = msg.content[0].text
    return json.loads(text[text.find("{"): text.rfind("}") + 1])  # tolerate stray prose

result = judge("banner_017.png",
               "Summer sale banner, 1200x628. Headline exactly 'Summer Sale, 30% off'. "
               "Show a beach towel and sunglasses. Brand colours teal and white.")
passed = result["gate"]["pass"] and all(
    result[k]["score"] >= 4 for k in ("adherence", "layout", "brand", "quality"))
print(passed, result)

Layer 3: reference comparison

Some tasks have a reference, even in generation. Image editing is the main one: given an original and an instruction, you can compare against a human-made target edit, check that unedited regions are preserved (pixel or LPIPS difference inside the unchanged mask), and check the edit itself with a VQA question ("Is there a car in the image?" should now be "no"). Brand work has a style guide: compare logo crops against the official logo, and extracted colors against the palette. Design reproduction ("make this mock-up as a web page") can be graded by rendering the result and comparing it with the target screenshot, the same idea as screenshot-based checks in VisualWebArena and SWE-bench Multimodal (Modules 07 and 08).

Layer 4: human review

Humans stay in the loop for three jobs: labeling the calibration set for the judge, reviewing a random sample of production outputs every week (Module 15), and adjudicating disagreements between judge and code checks. Pairwise comparisons ("is version B's output better than version A's?") are faster and more consistent for reviewers than absolute scores, exactly as in arenas.

Tying into computer use

Everything in Part A applies when the agent's "image" is a screenshot. A computer-use agent that misreads a small button label has a perception failure, not a planning failure, and resolution and cropping decide how often it happens (ScreenSpot-style grounding benchmarks in Module 08 measure exactly this). Everything in Part C applies when the agent's output is visual: a web-building agent's result should be rendered and graded like an image. In practice, many teams evaluate front-end and design agents with the four layers above, applied to screenshots of the rendered output.

Putting it together: choosing metrics

Check yourself

Sources and further reading

• Yue et al., MMMU (2023) and MMMU-Pro (2024).
• Lu et al., MathVista (ICLR 2024).
• Masry et al., ChartQA (ACL Findings 2022).
• Mathew, Karatzas, Jawahar, DocVQA (WACV 2021).
• Fu et al., Video-MME (CVPR 2025); leaderboard at video-mme.github.io.
• Li et al., MVBench (CVPR 2024).
• Chen et al., Are We on the Right Way for Evaluating Large Vision-Language Models? (MMStar) (2024).
• Zhang et al., Long Context Transfer from Language to Vision (LongVA, V-NIAH) (2024).
• Heusel et al., GANs Trained by a Two Time-Scale Update Rule... (FID) (2017); Jayasumana et al., Rethinking FID (CMMD) (CVPR 2024).
• Hessel et al., CLIPScore (EMNLP 2021); open_clip.
• Ghosh, Hajishirzi, Schmidt, GenEval (NeurIPS 2023 D&B).
• Huang et al., T2I-CompBench (NeurIPS 2023).
• Hu et al., TIFA (ICCV 2023); Cho et al., Davidsonian Scene Graph (DSG) (ICLR 2024); Lin et al., VQAScore and GenAI-Bench (2024).
• Huang et al., VBench (CVPR 2024) and VBench++.
• Preference models: HPSv2, ImageReward, PickScore / Pick-a-Pic. Arenas: LMArena, Artificial Analysis; Singh et al., The Leaderboard Illusion (2025).
• Anthropic, Vision documentation (image content blocks, resizing limits; fetched September 2026).
• C2PA and Content Credentials.