Quorum · a small on-device study

How should a small model spend its thinking?

An AI model can often get a better answer if you let it do more work: ask it the same question a few times, or let it think for longer. Both cost computing power. I wanted to know where that extra work actually helps and where it just burns time, so I measured it on a normal laptop. No data center, no paid API.

+17pts
more answers right when it samples a few times instead of once
~⅓
the work for the same accuracy, spent only where it's needed
~512tok
where extra thinking stops helping
16GB
one MacBook, nothing in the cloud

Why I built this

I wanted to see it for myself

I'm a student, and I built this to understand it firsthand instead of taking it on faith. People kept saying you can make a small model answer better by giving it more compute when it responds, and that past a point more thinking stops helping. It sounded right, but I had not actually seen it, so I ran it on the laptop I already own and measured it. No cluster, no budget.

Everything here is what I found, including the parts where my own idea was not the best one. I tried to be honest about what the numbers do and do not show, because the point was to learn from it, not to win.

Start here

The 60-second version, in plain words

If this is all new to you, here is everything you need to follow the rest of the page. No background needed.

Reasoning model

An AI that writes out its steps before giving a final answer, the way you show your work on a math problem. The one I used is small (1.5 billion settings inside it), small enough to run on a laptop.

Greedy vs. sampling

"Greedy" means ask once and take the single most likely answer. "Sampling" means ask a few times with a bit of randomness, so you get a few different tries at the same question.

Self-consistency

Sample a few answers, then keep the one that shows up most often, like a vote. More votes usually means more right answers, but also more work. This is the well-known trick the project builds on.

Test-time compute

The fancy name for spending extra work when the model answers (more samples, or longer thinking) to get a better answer, without retraining it. Using that work well is the whole question here.

GSM8K

A standard set of grade-school math word problems people use to test reasoning. Each has one correct number, so a computer can check the answers.

Overthinking

Making the model think for a set number of tokens (a token is a chunk of text, about a word) and changing that number. "Overthinking" is the point where letting it go longer stops making the answer any better.

The whole study is one question: a small model can spend more work to do better, so how do you spend it without wasting it?

The idea

Spend more only on the questions that need it

Easy questions get the same answer after two or three tries. Hard ones come back different every time. So instead of giving every question the same budget, the method takes a few samples, checks how much they agree, and only keeps going when they disagree. How much the early answers agree is a free hint for "this one is hard."

1
Take 4 samples
→
2
Do they agree?
→
Agree · stop at 4 (cheap)
Disagree · take up to 16
→
3
Take the vote

Every method below reads the exact same set of pre-made samples, so any difference in accuracy comes from how the work is spent, not from one method getting luckier draws than another.

Findings

Four results, and what each one means

01

Sampling helps a lot, fast

Answering once gets 68% right. Voting over several answers gets about 85%, a 17-point jump. But voting over 16 is no better than over 8, so the help shows up early and then levels off.

Means: more samples help, but only up to a point.

02

Being picky gets there for less

Instead of taking 16 samples on every question, take a few and only keep going on the ones where the answers disagree. You end up just as accurate, using about a third of the samples.

Means: same accuracy, most of the cost cut.

03

Longer thinking levels off

Accuracy climbs fast as the model is allowed to think up to about 512 tokens, then flattens. Four times that budget buys only six more points.

Means: past a point, more thinking is wasted work.

04

Stopping early is safe

When the first 4 answers all agree, the final answer is right about 90% of the time. So stopping there to save work almost never costs you the right answer, which is exactly what makes the picky method safe.

Means: you can stop early on the easy ones without losing accuracy.

Try it yourself

Watch it spend the work, question by question

Each square below is one question. The number on it is how many samples that question got; green means the final answer was right, red means wrong. Drag the slider: a low setting makes it stop early on more questions (cheaper), a high setting makes it cautious and take more samples (more expensive). The numbers update live, worked out in your browser from the real data.

0.60

One square per question · number = samples spent · green right, red wrong · hover for detail

Where the picky method (blue star) lands against the simple ones (grey dots). Up and to the left is better: more accuracy, fewer samples.

Being honest

Did my rule win? No, and that's the point

There is more than one way to decide when the early answers agree enough to stop. I wrote a simple one. There is also a known method from a 2023 paper (a stats-based rule from Adaptive-Consistency by Aggarwal et al.). I tested both against always taking 16 samples, and both reached the same accuracy for a fraction of the work. That is the result that matters. The known rule gets there on a little less work than mine, so I am saying that plainly instead of pretending my version won.

Two more things worth saying out loud. Matching the accuracy of "always take 16" is partly built in: any question whose first few answers already agree gets sent to the same vote, so equal accuracy is close to automatic. The real claim is the saving, not the accuracy. And to make sure the saving isn't just a fluke of these exact 50 questions, I set the rule up on one half of them and then tested it on the other half it had never seen. It still used about 37% of the samples.

All three reach the same accuracy. What changes is how many samples they spend to get there, and fewer is better.

Overthinking

More thinking stops helping before the model stops

Here I make the model think for a set number of tokens, then force it to answer, and I try a whole range of limits. Accuracy rises fast at first, then flattens around 512 tokens. After that, letting it think longer barely changes the answer. At this size it does not clearly get worse, it just stops paying off.

Accuracy vs. how long the model is allowed to think (in tokens)

Does the trick really work?

Is "the answers agree" a real signal, or luck?

The method bets that when the early answers agree, the question is easy and the answer is right. Does that bet hold? Partly, and I want to be careful about it.

The dependable part: when the first 4 answers fully agree, the final answer is right about 90% of the time. That is what the method actually relies on when it stops early, and it is solid.

The part I will not oversell: if you measure the general link between how much the early answers agree and whether the answer is right, it comes out weak (a correlation of 0.18, with a range of -0.06 to 0.52 that includes zero). At only 50 questions, that is too shaky to call a real effect, so I do not lean on it. Two quick side notes: running low on thinking room barely predicts anything on its own (-0.04), so agreement is not just flagging long questions; and on the cleaner half of the questions the link looks stronger (0.64), but that is a small, after-the-fact slice, so I treat it as a hint, not proof.

The files

Code and full write-up

What this is not

Honest limits

Parsa Ahmadnezhad

Parsa Ahmadnezhad

Mathematics + Machine Learning @ University of Waterloo · 2030

I built and ran this on my own Mac to figure out where spending more work at answer-time actually pays off on a small reasoning model, and where it just burns tokens. I did it mainly to understand how it works.