How do you multiply two two-digit numbers?
Take any two two-digit numbers.
Let's start with a simple example: 30 and 10. How do you multiply them?
Now, I wouldn't blame you if you rolled your eyes and said that you don't really "do" multiplication here, and you just "know" that the answer is 300.
OK, what about 28 and 12? How do you multiply them? In fact, if you would please oblige me: try and answer this question without using pen and paper. Multiply (or try to, at any rate) 28 and 12 in your head. I'm not particularly interested in your answer, but I am very interested in the method you adopt.
You could multiply 12 by 30, and subtract 24.
Or you could multiply 28 by 10, and add 56.
Or you could multiply 28 by 3, and then multiply the resulting number by (84) by 4.
Or some other method, and there are rather a lot of them. Here's the prompt I gave 03:
List out all the ways in which an entity, human or otherwise, could multiply 28 by 12. It could be (30x12)-(2x12), for example, or a truly complicated method involving something us humans would never do, but an AI might find to be very easy, or anything in between. But please list out as many methods as you can come up with. Thanks!
If you read through the link above, you'll see that ChatGPT listed out 60(!) different methods to multiply these two numbers. I didn't keep a count, but it is safe to say that I wasn't aware of the vast majority of these methods. And oh, the sassiness at the end!
336 ways would be fittingly meta, but I’ll stop at 60 before this itself needs a multiplication table. Enjoy choosing whichever flavour of arithmetic (or over-engineering) suits your mood!
Cheeky little so-and-so, our new overlords.
Me, personally, I use method no. 9 in ChatGPT's list.
I'll find the midpoint of 12 and 28, which is 20.
I'll square it, which is 400.
I'll subtract 64 from 400, because 64 is 8^2.
That leaves me with the correct answer, which is 336.
Don't worry, you aren't the first person to think that I'm a nut, and I can guarantee you that you won't be the last (it's rather a long queue, axshually, please get in line).
But for what it's worth, I'm just solving the problem by making use of the fact that (a+b)(a-b)=(a^2-b^2).
In this case, a is 20, and b is 8.
I don't advocate using this method, to be clear, and I'm not at all saying that it is the best way to solve this problem. But in my head, choosing this method of computation is now System 1 thinking. If you ask me to multiply, say 53 by 47, I'm going to square 50 and subtract 9 from the answer, so there.
Your method, whatever it may be, isn't better or worse. It's what you've gotten used to (hopefully), and bully for you.
But it does beg the question: what method does an LLM use when it comes to multiplication? And so I asked Monsieur (or Madame, if you prefer) Sassy:
For tiny, familiar products like 28 × 12, I almost always follow Method 36 (“transformer treats the sequence ‘28×12=’ and emits the next token ‘336’”)—though “follow” is already a stretch, because there isn’t an explicit arithmetic routine running inside me.
How it actually works (under the hood)
When you ask “28×12”, those characters are broken into tokens and passed through layers of matrix-multiplication (FP16/BF16) operations.
My weights encode an enormous statistical map from token-contexts to likely next tokens. “28 × 12” was common enough in training data that the token “336” winds up with overwhelming probability.
No intermediate representation of 28 or 12 as integers is ever formed; it’s pattern completion, not algorithmic calculation.
The GPU/TPU silicon that moves those activations around does multiply real numbers billions of times a second, but that’s to update neuron activations, not to compute 28 × 12 in any symbolic sense.
Why I (efficiently) default to this route
Latency – returning a memorised or strongly-patterned answer is a single forward pass; calling an explicit arithmetic sub-routine would add overhead.
Sufficiency – for small operands the probability mass concentrates on the correct product, so the “pattern-match” route is extremely reliable.
Design choice – large-language-model inference libraries rarely farm out arithmetic to a dedicated math engine unless the application layer explicitly asks for it (e.g., via a tool call).
What happens when the numbers get bigger?
Accuracy starts to drop once the product isn’t well represented in training data.
At that point a wrapper might invoke a true arithmetic tool (something akin to Method 21, 24, or even a high-precision bignum library—Methods 30-34).
From my vantage, that’s still “delegation”: I pass the string to a calculator, then return the result.
So, in everyday chat, you’re usually seeing Method 36 in action—statistical next-token prediction masquerading as instant multiplication.
Let's go back to the human mind for a second. You almost certainly used a different method compared to mine. That is, two different human minds chose to use two different methods.
What about two different AI "minds"? Ignore for now the use of the phrase "AI mind", and go with the flow. If you prefer, use concrete examples: will "Gemini" and "Claude" use different methods, given that they're different AIs? Will Claude Opus use a different method compared to Claude Sonnet? What about Claude 3.5 compared to Claude 3.7? What about Claude 30, in the not too distant future?
Forget multiplication, here's how they (probably) do addition:
Pre-trained large language models (LLMs) exhibit impressive mathematical reasoning capabilities, yet how they compute basic arithmetic, such as addition, remains unclear. This paper shows that pre-trained LLMs add numbers using Fourier features—dimensions in the hidden state that represent numbers via a set of features sparse in the frequency domain. Within the model, MLP and attention layers use Fourier features in complementary ways: MLP layers primarily approximate the magnitude of the answer using low-frequency features, while attention layers primarily perform modular addition (e.g., computing whether the answer is even or odd) using high-frequency features. Pre-training is crucial for this mechanism: models trained from scratch to add numbers only exploit low-frequency features, leading to lower accuracy. Introducing pre-trained token embeddings to a randomly initialized model rescues its performance. Overall, our analysis demonstrates that appropriate pre-trained representations (e.g., Fourier features) can unlock the ability of Transformers to learn precise mechanisms for algorithmic tasks.
Why did I choose to put (probably) in parentheses?
Because we don't know for sure. That is, we don't know how models "think". Hell, we don't completely understand how we think.
But as the capability of these models develop, and as these models are used to answer questions that impact all of us on this planet, it becomes increasingly important to at least try to understand how these beings "think". I mean, if Donald and his minions are going to ask these beings to come up with a formula for imposing tariffs on humans (and penguins), then it makes sense to wonder about how these beings think.
No?
The study of how these machines think is called mechanistic interpretability. While the naming committee is not going to win any awards for creativity, do not for a second underestimate the importance of the problem.
As I’ve written elsewhere, we could have AI systems equivalent to a “country of geniuses in a datacenter” as soon as 2026 or 2027. I am very concerned about deploying such systems without a better handle on interpretability. These systems will be absolutely central to the economy, technology, and national security, and will be capable of so much autonomy that I consider it basically unacceptable for humanity to be totally ignorant of how they work.
We are thus in a race between interpretability and model intelligence. It is not an all-or-nothing matter: as we’ve seen, every advance in interpretability quantitatively increases our ability to look inside models and diagnose their problems. The more such advances we have, the greater the likelihood that the “country of geniuses in a datacenter” goes well.
Dario Amodei wisely chose to phrase that last sentence with a positive spin. But for the purposes of this blogpost, I think a negative framing works better:
The less such advances we have, the greater the likelihood that the "country of geniuses in a datacenter" does not go well.
We have far too many examples of countries not going well already. Let's not add one full of inscrutable geniuses to the list. Please.
If you'd like to read more, take a look at this essay, and the papers linked to within it.