An AI model can often get a better answer if you let it do more work: ask it
the same question a few times, or let it think for longer. Both cost computing
power. I wanted to know where that extra work actually helps and where it just
burns time, so I measured it on a normal laptop. No data center, no paid API.
+17pts
more answers right when it samples a few times instead of once
~⅓
the work for the same accuracy, spent only where it's needed
I'm a student, and I built this to understand it firsthand instead of taking it
on faith. People kept saying you can make a small model answer better by giving
it more compute when it responds, and that past a point more thinking stops
helping. It sounded right, but I had not actually seen it, so I ran it on the
laptop I already own and measured it. No cluster, no budget.
Everything here is what I found, including the parts where my own idea was not the
best one. I tried to be honest about what the numbers do and do not show, because
the point was to learn from it, not to win.
Start here
The 60-second version, in plain words
If this is all new to you, here is everything you need to follow the rest of the
page. No background needed.
Reasoning model
An AI that writes out its steps before giving a final answer, the way you
show your work on a math problem. The one I used is small (1.5 billion
settings inside it), small enough to run on a laptop.
Greedy vs. sampling
"Greedy" means ask once and take the single most likely answer. "Sampling"
means ask a few times with a bit of randomness, so you get a few different
tries at the same question.
Self-consistency
Sample a few answers, then keep the one that shows up most often, like a
vote. More votes usually means more right answers, but also more work. This
is the well-known trick the project builds on.
Test-time compute
The fancy name for spending extra work when the model answers (more samples,
or longer thinking) to get a better answer, without retraining it. Using that
work well is the whole question here.
GSM8K
A standard set of grade-school math word problems people use to test
reasoning. Each has one correct number, so a computer can check the answers.
Overthinking
Making the model think for a set number of tokens (a token is a chunk of
text, about a word) and changing that number. "Overthinking" is the point
where letting it go longer stops making the answer any better.
The whole study is one question: a small model can spend more work to do better,
so how do you spend it without wasting it?
The idea
Spend more only on the questions that need it
Easy questions get the same answer after two or three tries. Hard ones come back
different every time. So instead of giving every question the same budget, the
method takes a few samples, checks how much they agree, and only keeps going when
they disagree. How much the early answers agree is a free hint for "this one is
hard."
1
Take 4 samples
→
2
Do they agree?
→
Agree · stop at 4 (cheap)
Disagree · take up to 16
→
3
Take the vote
Every method below reads the exact same set of pre-made samples, so any
difference in accuracy comes from how the work is spent, not from one method
getting luckier draws than another.
Findings
Four results, and what each one means
01
Sampling helps a lot, fast
Answering once gets 68% right. Voting over several answers gets about 85%, a
17-point jump. But voting over 16 is no better than over 8, so the help shows
up early and then levels off.
Means: more samples help, but only up to a point.
02
Being picky gets there for less
Instead of taking 16 samples on every question, take a few and only keep going
on the ones where the answers disagree. You end up just as accurate, using
about a third of the samples.
Means: same accuracy, most of the cost cut.
03
Longer thinking levels off
Accuracy climbs fast as the model is allowed to think up to about 512 tokens,
then flattens. Four times that budget buys only six more points.
Means: past a point, more thinking is wasted work.
04
Stopping early is safe
When the first 4 answers all agree, the final answer is right about 90% of the
time. So stopping there to save work almost never costs you the right answer,
which is exactly what makes the picky method safe.
Means: you can stop early on the easy ones without losing accuracy.
Try it yourself
Watch it spend the work, question by question
Each square below is one question. The number on it is how many samples that
question got; green means the final answer was right, red means wrong. Drag the
slider: a low setting makes it stop early on more questions (cheaper), a high
setting makes it cautious and take more samples (more expensive). The numbers
update live, worked out in your browser from the real data.
One square per question · number = samples spent · green right, red wrong · hover for detail
Where the picky method (blue star) lands against the simple ones (grey dots). Up and to the left is better: more accuracy, fewer samples.
Being honest
Did my rule win? No, and that's the point
There is more than one way to decide when the early answers agree enough to stop.
I wrote a simple one. There is also a known method from a 2023 paper (a stats-based
rule from Adaptive-Consistency by Aggarwal et al.). I tested both against always
taking 16 samples, and both reached the same accuracy for a fraction of the work.
That is the result that matters. The known rule gets there on a little less work
than mine, so I am saying that plainly instead of pretending my version won.
Two more things worth saying out loud. Matching the accuracy of "always take 16"
is partly built in: any question whose first few answers already agree gets sent
to the same vote, so equal accuracy is close to automatic. The real claim is the
saving, not the accuracy. And to make sure the saving isn't just a fluke of these
exact 50 questions, I set the rule up on one half of them and then tested it on the
other half it had never seen. It still used about 37% of the samples.
All three reach the same accuracy. What changes is how many
samples they spend to get there, and fewer is better.
Overthinking
More thinking stops helping before the model stops
Here I make the model think for a set number of tokens, then force it to answer,
and I try a whole range of limits. Accuracy rises fast at first, then flattens
around 512 tokens. After that, letting it think longer barely changes the
answer. At this size it does not clearly get worse, it just stops paying off.
Accuracy vs. how long the model is allowed to think (in tokens)
Does the trick really work?
Is "the answers agree" a real signal, or luck?
The method bets that when the early answers agree, the question is easy and the
answer is right. Does that bet hold? Partly, and I want to be careful about it.
The dependable part: when the first 4 answers fully agree, the final answer is
right about 90% of the time. That is what the method actually relies on when it
stops early, and it is solid.
The part I will not oversell: if you measure the general link between how much the
early answers agree and whether the answer is right, it comes out weak (a
correlation of 0.18, with a range of -0.06 to 0.52 that includes zero). At only 50
questions, that is too shaky to call a real effect, so I do not lean on it. Two
quick side notes: running low on thinking room barely predicts anything on its own
(-0.04), so agreement is not just flagging long questions; and on the cleaner half
of the questions the link looks stronger (0.64), but that is a small,
after-the-fact slice, so I treat it as a hint, not proof.
It's small. 50 questions. The sampling and picky-method
results are averaged over three runs; the rest are from one run. Treat small
differences as noise.
One test set. The point where thinking levels off is specific
to GSM8K. Harder problems would likely move it, and could turn "stops helping"
into "actively hurts."
A small model. A 1.5B model will not top any leaderboard. The
point is spending work well, not raw accuracy.
Parsa Ahmadnezhad
Mathematics + Machine Learning @ University of Waterloo · 2030
I built and ran this on my own Mac to figure out where spending more work at
answer-time actually pays off on a small reasoning model, and where it just
burns tokens. I did it mainly to understand how it works.