Can AI models spot a 1px UI change? We tested
Frontier vision models flagged 93–99% of one-property UI changes in 800×600 screenshots but 64–86% on full pages. A pixel diff mask got all four to 100%.

Frontier vision models can spot a 1px UI change in a small screenshot, but they miss many on full pages. In our visual regression test, Claude Opus 5.5, Claude Sonnet 5.5, GPT-6 Astra and GPT-6.1 Sol flagged 93–99% of one-property UI changes between two 800×600 component screenshots, with no false alarm on identical pairs, while Claude Haiku 4.5 flagged 47%. On full-page screenshots about 6,000 pixels tall, those four models caught 64–86% of the subtle changes, and with a pixel diff mask added they caught all of them: let a pixel diff find what changed, and let the model explain it.
How did we test AI models on UI changes?
We generated pairs where the answer is known. Eight components (card, sign-in form, navbar, pricing grid, table, modal, settings panel, toast stack) were rendered with Playwright 1.63.0's Chromium 153 on macOS at 800×600, device scale factor 1. Each pair differs by exactly one change, most of them in two levels: "hard", the smallest realistic step, and "easy", clearly visible:
| Change | Element | Hard | Easy |
|---|---|---|---|
| Text color | Text | #374151 to #4b5563 | #374151 to #b91c1c |
| Background | Button | #4f46e5 to #4338ca | #4f46e5 to #16a34a |
| Padding | Button | 8px 16px to 9px 17px | 8px 16px to 14px 24px |
| Font size | Text | 15px to 16px | 15px to 19px |
| Font weight | Button | 400 to 500 | 400 to 700 |
| Corner radius | Container | 8px to 6px | 8px to 0 |
| Position | Button | 1px to the right | 12px to the right |
| Line height | Text | 1.5 to 1.6 | 1.5 to 2.1 |
| Hidden badge | Badge | visibility: hidden | |
| One word | Text | "breathable" to "waterproof" |
That makes 144 changed pairs, plus 24 identical pairs where the same component was captured twice. Three hard changes are one step on Tailwind CSS v3's scales: gray-700 to gray-600, indigo-600 to indigo-700, rounded-lg to rounded-md. pixelmatch at threshold 0 confirmed that every changed pair differs (99 to 41,650 pixels) and every identical pair matches:
// generate.mjs (excerpt): @playwright/test 1.63.0, pixelmatch 7.2.0, pngjs 7.0.0
const CHANGES = [
{ category: "text-color", level: "hard", css: ".t-text { color: #4b5563; }" },
// ...17 more, as in the table above
];
async function shoot(page, component, change) {
await page.setContent(html(component)); // component markup and base CSS
if (change?.css) await page.addStyleTag({ content: change.css });
await page.evaluate(() => document.fonts.ready);
return page.screenshot({ type: "png" });
}
// Pixel count and diff mask: changed pixels in red over a faded BEFORE
const pa = PNG.sync.read(before);
const pb = PNG.sync.read(after);
const out = new PNG({ width: pa.width, height: pa.height });
const count = pixelmatch(pa.data, pb.data, out.data, pa.width, pa.height, {
threshold: 0,
includeAA: true,
alpha: 0.2,
diffColor: [255, 0, 0],
});
The full-page set puts the same components mid-page among filler sections, captured with fullPage: true at 1280 pixels wide and 5,626 to 6,031 tall: the 64 hard changes plus 8 identical pairs.
Each model got the BEFORE and AFTER PNGs ("screenshots only"), or those plus the diff mask ("with mask"), and returned a JSON verdict through structured outputs: changed or not, one of 11 categories, the element, and a sentence. Claude ran through the official Anthropic SDK with default settings (no effort or thinking parameter), GPT through OpenAI's Responses API with detail: "high" images, one request per pair, on October 7, 2026. Every answer came from the model we asked.
Can Claude or GPT spot small UI changes in a screenshot?
On 800×600 crops, the four frontier models caught nearly everything:
| Model | Screenshots only | Hard (64) | Easy (80) | With diff mask | False alarms on 24 identical pairs (screenshots / mask) |
|---|---|---|---|---|---|
| Claude Opus 5.5 | 135/144 (94%) | 86% | 100% | 144/144 | 0 / 0 |
| Claude Sonnet 5.5 | 134/144 (93%) | 84% | 100% | 144/144 | 0 / 0 |
| GPT-6 Astra | 142/144 (99%) | 97% | 100% | 144/144 | 0 / 0 |
| GPT-6.1 Sol | 139/144 (97%) | 92% | 100% | 144/144 | 0 / 0 |
| Claude Haiku 4.5 | 67/144 (47%) | 6% | 79% | 128/144 (89%) | 0 / 1 |
All four caught every easy change, every 1px padding or position change, and every font-size and font-weight step. Their misses came from two changes: the 8px to 6px corner radius (Claude Opus 5.5 caught 2 of 8, Claude Sonnet 5.5 none, GPT-6 Astra 7, GPT-6.1 Sol 4) and the #374151 to #4b5563 text (6 or 7 of 8 each), plus one line-height change for Opus. The descriptions were precise: GPT-6 Astra wrote "The Add to cart button shifted 1 pixel to the right," and Sonnet 5.5 listed the button's edge coordinates before and after.
Claude Haiku 4.5, the oldest and cheapest model in the test, flagged 4 of the 64 hard changes and called the screenshots identical for every 1px move. With the mask it reached 89% but reported a change on one identical pair.
Why do models miss more changes on full-page screenshots?
With the same 64 hard changes on full pages, detection fell from 84–97% to 64–86%; no model flagged an identical page:
| Model | Screenshots only | With diff mask | Text color | Corner radius | 1px move |
|---|---|---|---|---|---|
| Claude Opus 5.5 | 47/64 (73%) | 64/64 | 1/8 | 0/8 | 8/8 |
| Claude Sonnet 5.5 | 41/64 (64%) | 64/64 | 0/8 | 0/8 | 4/8 |
| GPT-6 Astra | 55/64 (86%) | 64/64 | 1/8 | 7/8 | 8/8 |
| GPT-6.1 Sol | 44/64 (69%) | 64/64 | 0/8 | 1/8 | 5/8 |
The likely culprit is resizing, which both providers document. Claude Opus 5.5 and Sonnet 5.5 are on Anthropic's high-resolution tier: one token per 28×28-pixel patch, with images scaled down until the long edge fits 2,576 pixels and the cost 4,784 tokens. A 1280×6016 screenshot becomes 548×2576, 43% of its width, and our token counts match the documented resize rule exactly: 4,379 input tokens per comparison, two images at 1,840 tokens plus 699 for the prompt. For GPT-6 Astra, detail: "high" allows 2,500 patches of 32×32 pixels (OpenAI image input docs), so the same page shrinks to about 735×3456 (57%); GPT-6.1 Sol isn't in that table as of October 2026 but billed the same image tokens. Neither API resized the 800×600 crops.
At 43% scale, a 1px move becomes less than half a pixel, and thin text strokes blend with their background, diluting a one-step color change. Resolution isn't everything: GPT-6.1 Sol had GPT-6 Astra's pixel budget and missed 20 changes instead of 9, and a tall page gives the model more to scan. Anthropic's docs suggest cropping the region of interest for small targets; the diff mask did a similar job, pointing every model to all 64 changes.
Can the model tell what kind of change it was?
Mostly, on crops. Of the changes they flagged, the four frontier models named the exact category 93–97% of the time from screenshots and 92–99% with the mask, and the right group (color, geometry, shape, text style, content) 99–100%. Nearly every wrong name was a line-height change called padding or position, which is what a taller row looks like; Opus 5.5 noted that "padding versus line-height cannot be told apart from the screenshots." On full pages, exact naming fell to 53–87%: a 400 to 500 font weight widened the button by about a pixel, and Opus 5.5 called it padding on 7 of 8 pages.
The mask helped detection more than naming. It took every frontier model to 100% detection, but exact naming moved by 3 points at most on crops, and on full pages it ranged from a 19-point gain for Opus 5.5 (53% to 72%) to a 6-point drop for GPT-6 Astra (87% to 81%). It can also mislead: Sonnet 5.5 called 5 of the 8 one-pixel moves a radius change because the red outline traced the button's edges, and Haiku 4.5 read the red overlay as the interface's new color in 9 answers ("The 'Add to cart' button background color changed from blue to red."). Six of those still scored as the right category, so category scores can flatter a weak model.
Why do these numbers differ from the DiffSpot benchmark?
DiffSpot, published on arXiv in May 2026 by researchers at Tencent, found much lower numbers: across 3,900 single-property CSS changes, the best of 13 vision-language models, Gemini 3.1 Pro, identified 40.7%, Claude Opus 4.7 31.2% and GPT-5.4 30.5%, and every model stayed under 23% on the hardest tier. Its strongest models rarely invented changes either: Opus 4.7 and GPT-5.4 reported one on 0.4% of 500 unchanged pairs.
Our closest comparable figure, changes flagged with the right category, was 87–95% on crops. With 168 pairs we can't isolate the cause, but three differences each plausibly matter:
- Models: a newer generation, Claude Opus 5.5 and GPT-6 rather than Opus 4.7 and GPT-5.4.
- Task: DiffSpot asks for every visual difference in free text, judged on element, property and direction; ours answered yes or no, picked one of 11 categories, and were told changes can be tiny.
- Images: DiffSpot uses 1280×800 renders of real web pages and keeps only changes confined to one element; ours are single components on a plain background, and some of our changes push the content below them.
The hardest tiers line up: DiffSpot's is one Tailwind step, like our gray text and 8px to 6px radius, our most-missed changes too. On our busier full pages, "flagged with the right category" fell to 34–75%.
How much does AI screenshot review cost?
Images make up most of the input (65–95% of the tokens), and verdicts are short: a median of 41 to 69 output tokens on crops. Cost per 1,000 comparisons, from our measured tokens and Anthropic's and OpenAI's standard prices (checked October 7, 2026):
| Model | Input / output per 1M tokens | Crops | Crops, with mask | Full pages | Full pages, with mask | Median latency (crops / full pages) |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | $4 / $20 | $9.37 | $12.36 | $20.70 | $28.16 | 2.1 s / 3.6 s |
| Claude Sonnet 5.5 | $2 / $10 | $4.85 | $6.23 | $10.92 | $14.44 | 2.0 s / 3.1 s |
| GPT-6 Astra | $10 / $50 | $18.33 | $23.62 | $72.69 | $97.86 | 4.3 s / 7.6 s |
| GPT-6.1 Sol | $2 / $10 | $3.72 | $4.83 | $14.55 | $19.95 | 4.5 s / 6.3 s |
| Claude Haiku 4.5 | $1 / $5 | $2.03 | $2.73 | not run | not run | 1.9 s / not run |
A comparison used 1,975 input tokens on the Claude 5.5 models and 1,425 on GPT for crops, and 4,379 to 4,563 and 5,915 to 6,245 for full pages. Latency is the median per request from our machine, three requests in flight per run. A pixel diff runs locally, with no per-call fee.
When is a model alone enough?
A model on its own is a reasonable check when:
- One focused screenshot answers one question, such as an agent checking the component it just edited: on crops, the four frontier models caught 93–99% of changes with no false alarms.
- You have no baselines yet. The model needs only the two images you hand it.
- The change is a faint color that a tolerant diff ignores. Diff settings tuned to ignore rendering noise, Argos's default included, pass over a one-step gray like
#374151to#4b5563; each frontier model flagged 6 or 7 of those 8 on crops.
It isn't enough to guard a whole app: on full pages, 14–36% of the hard changes went unnoticed, we didn't measure how stable a verdict is across reruns, and a model's "no change" leaves no evidence to review. Haiku 4.5 missed 60 of the 64 hard changes even on crops.
Do you still need pixel diffing if an AI agent reviews your UI?
Yes. A pixel diff answers "did anything change?" deterministically: at zero tolerance, pixelmatch flagged all 208 changed pairs and none of the 32 identical ones, the same way every run. Models are better at the next question: what changed, and was it intended? The setup we'd use:
- Let the diff decide whether something changed, with a tolerance you chose: a threshold that hides anti-aliasing noise can also hide a one-step color (how thresholds behave in Playwright and Argos).
- Show the model the change, not the page: crop base and head around the mask's bounding box, or send the mask along. Cropping avoids the downscaling that likely hid changes on full pages.
- Ask the model to explain which element changed and whether that matches the pull request, then check its answer against the mask.
- Use a frontier model. Haiku 4.5 missed most hard changes and misread the mask.
- Keep a person as the approver: accepting a new baseline decides what your UI should look like.
Where does Argos fit?
Argos is an open-source visual regression testing platform for Playwright, Storybook, Vitest and Cypress, and it splits the work this way. Comparisons are deterministic pixel diffs computed with odiff on Argos's servers; no AI model decides whether a screenshot changed (how Argos detects visual differences; the reasoning is in AI visual testing: what the AI actually does). The Argos MCP server and CLI give an agent each changed screenshot with its base, head and diff image URLs, so it can crop around the mask and explain the change against the pull request, while a person approves in Argos or on the pull request:
npx @argos-ci/cli build snapshots https://app.argos-ci.com/acme/web/builds/42 --needs-review --json
Setup for Claude Code, Cursor and Codex is in how to let coding agents review your visual diffs; other vendors' integrations are compared in visual testing tools with an MCP server.
What Argos doesn't do: it doesn't run a vision model, so explanations come from the agent you connect, and its diff ignores anti-aliased pixels and faint color noise by default. On our crops, Argos's open-source diff logic at the default threshold of 0.5 passed over the one-step text and button colors and the radius changes on our low-contrast cards; at 0.1 it flagged every color change, though some radius changes still passed. Lower threshold where such changes matter (Playwright SDK reference).
What are the limits of this test?
- Synthetic UI: simple components on a plain background, one change at a time. The full-page results suggest denser pages make detection harder.
- One browser, one density: Chromium 153 on macOS at a device scale factor of 1. At 2x, a change covers four times the pixels, and large screenshots shrink more.
- One run per pair, 8 pairs per change type: one answer moves a cell by 12.5 points, and rerun stability is unmeasured.
- Default settings: no effort, thinking or temperature parameter. More reasoning might find more, at a cost.
- A helpful prompt: "Changes can be very small: a slightly different color, 1px of spacing or position, a slightly different font size or weight, a changed corner radius." With that hint and a category list, these numbers are closer to a best case than to an open-ended "does this look right?" review.
- A fixed-position modal, which sat in the first 800 pixels of its full page instead of mid-page.
- No Gemini or open-weight models.
How can you rerun this test?
The Claude side of the runner, with @anthropic-ai/sdk 0.131.0 and zod 4.6.5 on Node.js 24.15.0:
// run.mjs (excerpt): one request per pair
import { readFileSync } from "node:fs";
import Anthropic from "@anthropic-ai/sdk";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";
import { z } from "zod";
const CATEGORIES = [
"text-color",
"background-color",
"padding",
"font-size",
"font-weight",
"border-radius",
"position",
"line-height",
"removed-element",
"text-content",
"none",
];
const Verdict = z.object({
changed: z
.boolean()
.describe("true if anything visible differs between BEFORE and AFTER"),
category: z.enum(CATEGORIES).describe('the kind of change, or "none"'),
element: z.string().describe("which element changed, or an empty string"),
description: z.string().describe("one short sentence"),
});
const SYSTEM =
"You review visual regression test results for a web app. You get two screenshots of the same UI, " +
"BEFORE and AFTER a code change, rendered by the same browser at the same size. Decide whether anything " +
"visible changed. Changes can be very small: a slightly different color, 1px of spacing or position, a " +
"slightly different font size or weight, a changed corner radius. Do not guess: if you see no difference, " +
'answer changed=false and category "none".';
const INSTRUCTIONS =
"Compare BEFORE and AFTER. Answer with: changed (boolean), category (one of " +
CATEGORIES.join(", ") +
"), element (which element changed), description (one short sentence).";
const DIFF_NOTE =
"DIFF MASK: pixels that differ between BEFORE and AFTER are drawn in red over a faded copy of BEFORE. " +
"If the mask has no red pixels, nothing changed.";
const image = (path) => ({
type: "image",
source: {
type: "base64",
media_type: "image/png",
data: readFileSync(path).toString("base64"),
},
});
const client = new Anthropic();
const response = await client.messages.parse({
model: "claude-opus-5-5",
max_tokens: 16000,
system: SYSTEM,
messages: [
{
role: "user",
content: [
{ type: "text", text: "BEFORE:" },
image("before.png"),
{ type: "text", text: "AFTER:" },
image("after.png"),
// "with mask" only: { type: "text", text: DIFF_NOTE }, image("diff.png"),
{ type: "text", text: INSTRUCTIONS },
],
},
],
output_config: { format: zodOutputFormat(Verdict) },
});
if (response.stop_reason === "refusal") throw new Error("refused");
console.log(response.model, response.parsed_output, response.usage);
The OpenAI version sends the same parts as input_image items with detail: "high" to client.responses.create() (openai 7.28.0), with the schema as a strict json_schema in text.format (structured outputs). Score each answer by comparing changed and category with the change you injected.
FAQ
Can AI detect visual regressions from screenshots?
Partly. In our October 2026 test, Claude Opus 5.5, Claude Sonnet 5.5, GPT-6 Astra and GPT-6.1 Sol flagged 93–99% of one-property UI changes between two 800×600 screenshots with no false alarms, but 64–86% of subtle changes on full-page screenshots. A pixel diff caught all of them, so use the diff to detect regressions and the model to explain them.
Can Claude or GPT spot a 1px change?
On a small screenshot, yes: all four frontier models flagged the eight 1px button moves on 800×600 crops and described each as a one-pixel shift to the right. On full pages, Claude Sonnet 5.5 caught 4 of 8 and GPT-6.1 Sol 5 of 8; Claude Haiku 4.5 missed all eight even on crops.
Is pixel diffing still needed if an AI agent reviews my screenshots?
Yes. The pixel diff is deterministic and caught every change in our test, while the models missed up to 36% of subtle changes on full pages, where both APIs shrink the screenshot before the model sees it. Give the agent the diff mask or crops around it: with the mask, every frontier model flagged every change.
Does a diff image help an AI model review UI changes?
For finding changes, a lot: with a red diff mask, the four frontier models flagged all 208 changes, up from 64–99% without it. For naming them, little: on crops, the share with the right category moved by 3 points at most. Claude Haiku 4.5 misread the red overlay as the interface's new color nine times.




