We let a cheap model take our quiz: 98.1%. So we stopped making quizzes

Between 9 and 22 October 2026 we pointed one of the cheapest text models we could find at our own question bank. It answered 255 of 260 sampled questions correctly. Its median time per answer was 652 milliseconds. The whole run cost $0.0033, about a third of a cent.

We had expected it to do well. We had not expected it to make the quiz pointless.

This post is about what that number means, what we changed because of it, and what it still does not prove.

What we measured

• Model: one cheap, text-only model (Claude Haiku 5.5), given each question and its answer options as plain text.
• Sample: 260 questions drawn from HIM's question banks.
• Result: 255 of 260 right (98.1%), median 652 ms per answer, $0.0033 for the whole run.
• Dates: 9-22 October 2026. An earlier run on 25 September 2026 gave 97.5% at about half a second per answer.
• Image questions: on a separate set of 40 image questions, the same cheap model got 92.5% and a larger model (Claude Sonnet 5.5) got 100%.
• Who measured: HIM, on our own questions. This is an internal measurement, not an independent study.

A right answer is no longer a human signal

A quiz rests on one assumption: if you pick the right option, you probably knew it. That assumption held for most of the history of quizzes, because the only way to the right answer was through a person's head.

It does not hold any more. A cheap text-only model answered 255 of 260 sampled HIM questions (98.1%), our own measurement, 9-22 Oct 2026. That is why HIM is never a plain quiz.

Think about what 652 milliseconds means. A person needs longer than that just to read a four-option question. So the model is not only right more often than most people. It is also faster than anyone can read, and it costs almost nothing to run thousands of times.

For a free trivia game, that might be a nuisance. For anything with a leaderboard, points, a season or a prize, it changes the game completely. A plain quiz with money attached would pay the fastest script, not the person who knows the most.

We ran the test twice, weeks apart, and got nearly the same answer both times (97.5% in September, 98.1% in October). It was not a fluke.

Pictures did not save the quiz either

The obvious next idea is to move the knowledge into images: a photo of a stadium, a crop of a map, a chart. We tried that too.

On 40 image questions, the cheap model got 92.5% right and a larger model got all 40. Images slow a cheap model down a little. They do not stop a better one.

Outside HIM, others see the same trend with the oldest human test on the web. In one vendor study, off-the-shelf AI agents solved reCAPTCHA v2 in up to 60% of trials. The study was small and run by a company that sells human checks, so treat it as a signal rather than a verdict, but the direction is clear: a test that only asks "can you see this and click the right thing?" is getting easier for machines every few months.

What we chose not to do

There is a tempting shortcut here: make the question hard to read. Blur it, animate it, break the text into noise that a person can just about decode and a model cannot.

We decided against it, for two reasons.

The first is people. Text that fights a model also fights anyone with poor eyesight, a screen reader, a small phone or a slow connection. A mind game that is hard to read is unfair before it starts.

The second is that it does not work for long. Anything we send to a browser can be read by a script in that browser. Hiding the question buys hours or days, not a moat.

So the rule at HIM is "detect, don't hide". Every question is readable by everyone and timed the same for everyone. We look for bots in how a game is played, not by making the game worse for humans.

What we built instead

We stopped asking "can we write questions a model cannot answer?" Some of those exist, and we measure them (more on that below), but most questions anyone can write, a model can answer. Instead we asked: what makes a round of play hard to fake at scale, and what can we measure honestly?

1. Questions run inside game mechanics

On HIM a question is never a bare list of four options. It runs inside a game mechanic: you might catch the right answer as it falls, estimate a number, put events in order, choose higher or lower, or hold your side in a tug of war. Creator leagues use 17 mechanics today.

A mechanic changes what a right answer looks like. It is a sequence of actions over time, not a single string that a model sends back. We have not yet published per-mechanic measurements of how a model does on each one; that is ongoing work, and when we have numbers we will publish them with dates. What we can say now is narrower: the plain format is the one we measured, and it failed.

2. The server keeps time

The question appears word by word, your clock starts when the answers fall, and points melt every 6 seconds. The server times every answer.

The browser does not decide how fast you were. That matters for honesty between players, and it gives us a record of how each answer was played, not just whether it was right.

3. A short human check before prize entry

A short human check decides prize entry. It does not establish who you are, and we do not describe it that way. It is a small cost for one person and a growing cost for anyone running many accounts with scripts. Prize games are for players 18 and older and are not available in every country.

4. The answer key stays on the server

Answer keys stay on the server until the round ends. While you play, an answer comes back as right or wrong; the full list of answers appears at the end of the round, for the people who played it. A script watching the browser learns no answers while the round is running.

5. Every question is scored against the AI first

This is the part that turned a bad result into a useful one.

Since the model is so cheap, we let it attack every question. Each new item gets three tries. We scored 24,042 items with ~72,000 AI attempts for $0.96 (HIM internal measurement, Oct 2026). Measuring the AI side of a question costs a fraction of a cent.

Most items fall: the model gets them right three times out of three. A small share do not. On 23 October 2026, HIM's public counter showed 12,516 items measured, 1,187 missed by our cheap AI attacker, 0 human-checked answers counted and 0 gold. The attacker is one cheap model; the missed count is expected to fall under stronger attackers.

Those 1,187 items are where the interesting work starts. An AI miss does not mean people know the answer. Sometimes the question is badly written, or the "right" answer is wrong. The only way to find out is to put the item in front of many people and count. That is what the counter is waiting for, and it is why "0 human-checked answers" is the most important number on the page right now.

At the end of each round, HIM marks in gold every question you got right that the AI missed in all three tries. That mark is the first, small piece of the gap made visible.

What this does not prove

It is easy to oversell a fix like this, so here is the honest list.

• Nothing here makes bots impossible. What we can do is measure bots and make them expensive. A determined team with a strong model and real people in the loop can still play.
• One cheap model is a weak attacker. The 98.1% and the 1,187 misses come from a single inexpensive model. A stronger model answers more, as the image test showed. We expect the missed count to fall when stronger attackers are added.
• The mechanics are not yet measured one by one. We know the plain format fails. We do not yet have published numbers for each mechanic.
• Outside traction is about zero. Every on-chain game so far is a team test, 0 human-checked answers have been counted and there is 0 gold. The design is live; the evidence from real players is not in yet.

The lesson, if you build quizzes

If your product asks a question in text with a few options and rewards the right one, assume a model can take it, quickly and for almost nothing. That is true for trivia apps, for training quizzes at work, for community competitions and for "prove you're human" puzzles.

You do not have to make the question unreadable to respond. You can change what you reward (how something was played, over time, on a clock the server keeps), keep the answers off the client, and measure your own items against a cheap model before people ever see them. That last step costs cents and tells you which of your questions are worth anything.

Play the round, then send it to a friend

Today's round is at him.finance. It takes a couple of minutes. The questions sit inside a game, the clock is the same for everyone, and at the end you will see which questions you knew that our AI missed. Sign in to keep your place on the board; connecting a wallet takes one signature, with no transaction and no fee.

Then challenge a friend from your result screen. With a challenge link they play the same questions and race your recorded run on the same clock. No points for sharing, nothing to win from the link. Just the question we keep asking: who knows more, you or the machine?