The Pelican Test: One Prompt, 66 Models, and What Each One Drew

One 8-word prompt—"Generate an SVG of a pelican riding a bicycle"—has been fed to almost every new model for two years: from 16 models in October 2024 to 66 models and 103 images today. It was never a serious benchmark; its creator even wrote a post specifically saying "don't use it to compare models." But it turns the question "can this model actually draw?" into something you can grasp at a glance—so this post lays out, in chronological order, what each vendor's model drew.

All works below come from public sources (Simon Willison's pelican benchmark and its mirror site pelicanzoo.ai). Each one is credited, and copyright belongs to the original authors.

The Bottom Line First

It isn't a benchmark, but it is the best-looking cross-section of this generation of model capabilities: the 2024 batch was "two circles and a triangle"; the 2026 batch is finished work you could drop straight into an illustration.

And it's a picky test: success here has nothing to do with taste—the model can't see what it drew and can't revise midway; it has to assemble a pile of coordinates and paths into "a bird riding a bike" in one shot. What it really measures is the ability to "generate a complex structure in one pass."

1. Why an 8-Word Prompt Went Viral

The creator's reasons for choosing this subject were simple: he likes pelicans (he lives by the coast), and he was confident that there could be no SVG of "a pelican riding a bicycle" in the training data.

More importantly, there are these three points:

1. No feedback loop—a language model can't see what it drew; it assembles shapes blindly from coordinates and radii, with no chance to "take a look and fix it" along the way;
2. Bicycles are hard to draw from memory—if you don't believe it, try drawing one right now; odds are the frame geometry will be wrong;
3. Pelicans are inherently the wrong shape to ride a bike—in the creator's own words: "Pelicans can't ride bicycles. They don't have the right shape."

A free bonus: models write comments in the SVG—<!-- wheels -->, <!-- frame -->, <!-- pelican legs -->… so you can compare "what it meant to draw" against "what it actually drew"—the gap is usually where the comedy is.

2. A Two-Year Timeline: Images + Numbers

2024-10 · First Test, 16 Models

Left: Claude 3.5 Sonnet (2024-10-22)—two spoked wheels, a simple frame and a small bird on top; surprisingly decent. Middle: Llama (Cerebras version)—blue background, green ground and a black-and-white blob in the middle; you can't make out a bird or a bike. Right: Gemini 1.5 Flash 8B—two yellow blobs; the creator gave it an "honorable mention."

Other memorable verdicts from the same batch: the earlier Claude 3.5 Sonnet (2024-06) drew "two circles and a triangle, with a beak like a yellow banana smiley face"; Qwen2.5-Coder's looked like a tractor; Mistral Small 3 was "a duck squatting on a barbell."

2025-06 · 34 Images, Ranked by Elo in Head-to-Head Matchups

Once there were 34 images, the creator didn't want to judge them himself anymore, so he had Claude write a head-to-head comparison tool: 560 matchups, at a total judging cost of about 18 cents, producing an Elo ranking. The champion was the Gemini 2.5 Pro preview; last place went to Llama 3.3 70B.

2026-02 · The First "Usable Illustration"

This one was described as an illustration you could use as-is—clean lines, a correct frame, and a bird with a throat pouch and eyes. Getting here from "two circles" took one year and four months.

2026-04-16 · A Small Model on a Laptop Beat a Frontier Model

Two models released on the same day: on the left, Qwen3.6-35B-A3B (a 20.9GB quantization running on a laptop) drew something clear and complete; on the right, Claude Opus 4.7 got the frame geometry wrong—and with thinking maxed out and another try, it was still wrong.

2026-05 · A Sweep of 21 Quantizations: All Broken, Regardless of Size

This grid shows 21 quantization levels of Granite 4.1 3B (each cell labeled with its GGUF size). Two conclusions: they're broken across the board, and there's no correlation with size—1.2GB did no better than 6.34GB, and ironically one of the very small ones (IQ2_M) produced a recognizable bicycle, even captioned "Pelican Rides!"

2026-09 · On the Same Grid, a 40× Spread in Per-Image Cost

Every cell in this grid is labeled with output tokens and cost: a single pelican ranged from 1.57 cents to 63.21 cents. Within the same family, GPT-6 Astra's low level already beat everything GPT-5.6 Sol produced (9.55 cents). In other words, the "gap" you see on a leaderboard is sometimes just someone else turning the level all the way up.

2026-09-01 · Maximum Thinking Level: 13 Minutes 54 Seconds, $3.30

The same model at 5 thinking levels; the maximum level clocked 13 minutes 54 seconds, 65,927 tokens, $3.30. It also added a blue hat to the pelican on its own initiative and put a fish in the bike basket—at this point the question isn't "does it look right" but "does it want to add its own flourishes."

3. What the "Bird on a Bike" Looks Like from Each Vendor in 2026

Left to right, top to bottom:

• Claude Sonnet 5: line drawing on a white background, the bird's body a gray block with an orange beak, the bicycle a detailed line sketch. It reads as a pelican, but more like a goose—the throat pouch is gone.
• GPT-5.5 (xhigh): the most "textbook" one—throat pouch, orange legs, blue frame, grass and clouds all present, plus speed lines for coasting.
• Kimi K3: the most design-conscious—red scarf, red frame, and a gray road with a dashed center line, with sun and clouds too. Someone has started giving it "styling."
• GLM-5.2: soft palette, black wheels, beige background; overall like a quiet vector illustration, though the bird's head is placed a bit awkwardly.
• Qwen3.8-27B (thinking): big throat pouch, red frame, green grass—clean and crisp.
• Qwen3.8-27B (thinking off): another version of the same model—the frame turns black, the sun gets bigger, the bird looks more like a stork. Similar style, different details.

To sum up this row in one sentence: by 2026, "can it draw this at all" is no longer the question; the question has become "whose style does it look like."

4. And Some Never Get It Right

Left: OLMo 3 32B—literally "two circles + a few geometric blocks" (a brown rectangle, two gray triangles); you can't make out a bird or a bike. Middle: GPT-5 nano—the bicycle skeleton is barely recognizable, and the bird is a white smudge. Right: Claude Opus 4.7 at max level—looks decent, but the creator's verdict was "the frame shape is completely wrong, just wrong in a different way."

Putting these three side by side shows one thing: the same test can separate failure modes at different scales—some can't draw (OLMo), some can draw but don't bother finishing (GPT-5 nano), and some draw something that looks good but is structurally wrong (Opus 4.7 max).

5. Three Counterintuitive Conclusions

1. The creator himself says "don't treat it as a benchmark." In the first year there really was a correlation of "draws a good pelican ≈ useful model"; now it's basically gone—GLM-5.2 draws a better pelican than the frontier models, but he doesn't consider GLM to be in that tier. More importantly, it can't measure what really matters today: long-conversation retention and tool-calling reliability.

2. The reasoning level often matters more than the differences between models. On the same grid, cost varies 40× and time ranges from 7 seconds to 13 minutes 54 seconds.

3. Quantization isn't the main factor; "the base model couldn't draw in the first place" is. In the Granite case, the base model was already near the capability floor, so no quantization scheme could show a difference—first make sure the base model is strong enough, then compare quantizations; otherwise you're measuring noise.

6. A Local 125B Ran Both Tracks Too

I ran the same test on a locally deployed Qwen3.8-Flash-Next (125B sparse MoE, IQ3_S quantization).

Classic track—the same prompt, asked three times:

16.2 seconds, 21.2 seconds, 15.3 seconds, with visibly different levels of polish across the three: the frame geometry was correct all three times; the differences were in the pelican itself (throat pouch, neck length, wings). In other words, it "understands how a bicycle is built," but has only a vague idea of "what a pelican looks like."

Animation track (the tougher prompt from the community comparison: spinning wheels + pedaling motion):

In 97.5 seconds it generated a complete 28,204-character HTML file: the HTML defines 7 sets of @keyframes (spin, roadMove, groundMove, drift…), each of the two wheels has its own wheel-spin element with a 360° rotation, the road moves and the clouds drift. It gave the scene a title, "El Pelícano Ciclista," and there's a speedometer in the top right showing 24 km/h.

On the leaderboard, this local result would count as upper-middle: it produces finished work and can animate, but don't expect it to be consistent every time—three samples of the same prompt already show clear variance. (Tuning details are covered in another post on this site.)

7. What This Test Measures, and What It Doesn't

It measures: under closed conditions—"can't see its own work, can't iterate"—whether a model can organize a large pile of coordinates and paths into a structurally correct finished piece. This is closer to the ability to "generate a large block of structured code in one shot" than to aesthetics.

It doesn't measure: long-conversation retention, tool-calling reliability, the edges of instruction following—that is, the things that really determine today whether a model "can get work done."

But it has one role nothing else can fill: it's the only test where "the result can be verified at a glance and saved as a file." One SVG is a reproducible piece of evidence—it contains coordinates, comments and structure, it can be quantified (tokens, time, cost), and you can see at a glance where it broke. With a leaderboard you have to trust its scores; with a pelican you can judge for yourself.

References

• Simon Willison's pelican collection (the original source, with commentary on every release and all the SVGs): https://simonwillison.net/tags/pelican-riding-a-bicycle/ | Official gallery repo: https://github.com/simonw/pelican-bicycle
• A mirrored collection of all 103 works (many images in this post come from here): https://pelicanzoo.ai/
• Community submissions (classic SVG / animated HTML tracks): https://pelicanbenchmark.com/
• Side-by-side comparison of animated SVGs from 105 models: https://rmohid.github.io/svg-model-compare/
• A Chinese-language roundup of same-prompt comparisons: 36Kr, "A Pelican Became the Biggest Star of the Agent Era"
• Image sources: the model works in the body are all captured from Simon Willison's pelican benchmark and its mirror site; copyright belongs to the original authors, and this post cites and credits each one. The two local images (classic three runs, three animation frames) are actual outputs from my own tests for this post
• Data notes: local-side timings/tokens were collected on 2026-10-15 on my own server (RTX 5090 / 62 GB RAM), single samples, not averaged; leaderboard-side figures are cited from the sources above, each from a different point in time