The Royal Swedish Academy of Sciences, which selects the Nobel Prize laureates in Physics on behalf of the Nobel Foundation, probably also recognized that artificial intelligence (AI) would have an enormous impact on the future of human society. Thus, the 2024 Nobel Prize in Physics was awarded to two scientists who made major contributions to the foundations of AI: Geoffrey Hinton of the University of Toronto and John Hopfield of Princeton University. In fact, OpenAI, a research laboratory founded in late 2015, had already begun exploring how to commercialize AI. In 2017, a landmark paper appeared: “Attention Is All You Need.” It introduced the concept of the Transformer architecture, which quickly became the dominant architecture in AI and was also adopted by OpenAI. After several years of research, OpenAI launched ChatGPT in November 2022. The “T” stands for “Transformer,” while “GP” stands for “Generative Pre-trained.” Its arrival changed the world, unleashing a wave of AI that swept across the globe. The idea that intelligence could emerge from computer calculations alone seemed almost unimaginable.
We all know that other living creatures—animals with body heat and the ability to move—possess some degree of intelligence. Compared with human intelligence, however, theirs seems insignificant. Now, computers—nonliving systems with no body heat or ability to move on their own—can also produce intelligence through complex calculations. The prospect that this intelligence could far surpass our own leaves many people deeply uneasy.
Most of us have now tried AI. When we have a question, we ask it, and it patiently provides a detailed answer. This is “knowing what happens”: we know that AI operates through computers and it answers our questions. But to “understand why it happens”—to understand how AI possesses such remarkable intelligence and can answer such a wide variety of questions intelligently—we must delve into the computer programs and calculations underlying it, seeking to understand why computation itself can give rise to intelligence.
What exactly is intelligence? According to a common definition, intelligence is the ability to learn, reason, understand, solve problems, and adapt to new situations. In a person who possesses these abilities, the internal processes involve learning, reasoning, and understanding, while their outward expression is the ability to solve problems and adapt to new situations.
It is generally accepted that human beings possess intelligence. Today, we will set aside the question of whether other species possess intelligence and focus instead on how a newborn baby acquires it (Figure 1). As we all know, babies develop intelligence as they grow through repeated cycles of learning, receiving feedback, and making adjustments, absorbing new experiences and developing day by day. Similarly, AI becomes intelligent through repeated cycles of learning—called “training”—receiving feedback, and making adjustments.
Today, I will describe how I designed a small AI model, which I will call “TinyLLM,” to understand how AI acquires intelligence. Before turning to this topic, let us first look at an example of a large commercial AI model of the kind we use.
How AI Gives You an Answer:
When you open an AI app such as ChatGPT, it asks what you would like to ask. You enter a question, for example: “What color is sky?” From the AI’s perspective, you have supplied four English words and a question mark. AI calls these words and punctuation marks “tokens,” so you have given it five tokens. How many tokens (its vocabulary) does a large AI model have? A GPT-3.5 model released by OpenAI in 2022, with 12,288 dimensions, has a vocabulary size of approximately 100,000 tokens, covering English, Chinese, Japanese, Korean, French, German, and other languages, along with various symbols. Now that you have supplied five tokens, AI begins processing them with one goal: to predict the next token. It does this by calculating the probabilities of all the vocabulary tokens (~100,000) through an enormous number of matrix operations. Generally, the token with the highest probability becomes the next token. With this new token added, the sequence now contains six tokens. AI processes these six tokens in the same way to obtain the next one. This continues until it produces a final token called “stop,” at which point it delivers the complete answer to you.
Thus, regardless of how many tokens your question contains, once AI receives your input, it gets to work calculating an answer. This is why some people say that AI is simple: all it does is determine the next token. It may sound simple, but if you understand what happens inside a model such as GPT-3.5—with 578 matrices containing a combined total of nearly 200 billion parameters—you realize that the amount of computation involved is staggering.
Why make AI so large, with hundreds of billions of parameters? OpenAI’s primary aim is to gather as much of humanity’s digitized knowledge as possible and use it to train its language models. Its scope encompasses the entire world and all of humanity. To cover such a vast range of subjects, the number of parameters rises into the hundreds of billions. When the training text is broken down into individual tokens, the total comes to approximately 300 billion tokens.
Such large AI models are developed for commercial use, and only major companies with substantial financial resources and large teams of software engineers can build them. After using their products, people cannot help but be impressed by these models’ breadth of knowledge, powerful capabilities, and extraordinary speed. Their generative capabilities can even create new AI music and videos—things that Google Search simply cannot do.
Through training, AI acquires intelligence and can therefore create something new. But, put plainly, AI consists of an enormous amount of computation. How, then, does intelligence emerge from computation?
Designing a Small AI—TinyLLM:
To understand how intelligence emerges from computation, I designed a small AI model with just eight dimensions. Called “TinyLLM,” it has only ten tokens: heart, umbrella, tree, bucket, monkey, apple, tiger, boat, fish, and glove. In addition to these ten tokens, TinyLLM has six matrices, containing a total of 424 parameters. This small AI performs the same basic task as ChatGPT: predicting the next token. I gave TinyLLM 40 training examples, each consisting of three tokens. The training rule is simple: when you see the first two tokens in an example, your “next token” should be the third. For example, in the first row of Figure 2, you see apple and monkey. Remember this pair carefully and your “next token” is glove. In the second row, you see tiger and tree. Remember this pair carefully, and your “next token” is fish. Training means running a computer program. I asked AI to help write the program in Excel VBA. Once the code had been checked and debugged, training began.
Each token is represented by a “vector” containing eight numbers (because the model has eight dimensions). Initially, these eight numbers are random. The parameters in each matrix are also initialized to random values.
For the first training example—apple, monkey, glove—the vectors for apple and monkey are processed through the matrices to calculate the probabilities of all ten tokens. After this calculation, whatever the resulting probabilities may be, the program goes back and makes small adjustments to each parameter in the matrices. It also adjusts the parameters in the three vectors for apple, monkey, and glove. These adjustments have just one goal: to make small changes to the parameters in a direction that reduces the prediction error.
The program then moves to the second example: tiger and tree. Their vectors are processed through the matrices, which have already been slightly adjusted, to calculate the probabilities of all ten tokens. The program then makes further small adjustments to the matrix parameters and to the vectors for tiger, tree, and fish, again seeking to reduce the prediction error.
This process continues until every training example has been processed. The program then calculates the overall error, measured by the “cross-entropy loss”. This completes the first training round, or “epoch.” If the overall error is too large, the program begins a second epoch using the adjusted parameters. Training continues in this way until the error falls below a chosen threshold. At that point, training is complete. If after certain number of epochs the loss does not reduce, or gets bigger and bigger, then it should be aborted and a new training session started. It should be noted that there may be many failed training sessions before landing on a successful training. Figure 3 shows the 40 examples I used to train this small AI.
Figure 4 shows the values of the 424 parameters in the six matrices obtained after a successful training session.
Notice that everything TinyLLM learned during training has “evaporated” into numbers distributed across its many parameters. The same is true of ChatGPT: all the English sentences from encyclopedias used in training, all the passages from Shakespeare’s works, and all the poetry of Li Bai and Du Fu have “evaporated” into numbers distributed across hundreds of billions of parameters. Nowhere in the entire model can you find a single English or Chinese sentence used during training.
TinyLLM’s Next-Token Prediction Ability—Memory:
The six matrices in Figure 4 constitute my small AI model, TinyLLM. A simple way to test how well it works is to use the training examples in Figure 3. For each row, we supply TinyLLM with the first and second tokens and see whether it can accurately predict the third.
The test results showed that TinyLLM correctly predicted the third token in every row. This indicates that TinyLLM has the ability to remember: it retained all 40 examples it encountered during training. Its memory is not stored at a particular address in the computer, but is distributed across its many parameters.
TinyLLM’s Next-Token Prediction Ability—Generalization and Intelligence:
In addition to the 40 training examples, I prepared 15 test examples, as shown in Figure 5. All 15 differ from the training examples, and TinyLLM has never seen them before. Consider row 41, for instance. The pairing of two umbrellas does not appear anywhere in the training set; it is something new. Has TinyLLM, through training on the 40 examples, grasped anything that would enable it to predict the next token in row 41?
The test results showed that it correctly predicted the third token in every row (Figure 6). This indicates that, after training, TinyLLM had acquired something beyond memory; otherwise, it would not have been able to answer all 15 questions correctly. We therefore say that TinyLLM has the ability to generalize: it can solve problems and adapt to new situations. This suggests that it possesses a certain degree of “intelligence.”
What Exactly Has TinyLLM Grasped Through Training?:
An observant reader who carefully examines the 40 training examples in Figure 3 may already have spotted clues and discerned some underlying rules. In fact, the first 40 examples and the remaining 15 are simply elementary-school addition problems in which the sum of two numbers does not exceed nine. Thus, 0 + 0 = 0, 0 + 1 = 1, 1 + 0 = 1, 1 + 1 = 2, 0 + 2 = 2, 2 + 0 = 2, …, 4 + 5 = 9, and 5 + 4 = 9; 55 examples in all.
The correspondence between numbers and tokens is shown in Figure 7: heart = 0, umbrella = 1, tree = 2, bucket = 3, monkey = 4, apple = 5, tiger = 6, boat = 7, fish = 8, and glove = 9. With this explanation, the two umbrellas followed by a tree in row 41 simply mean 1 + 1 = 2. The monkey, apple, and baseball glove in row 55 mean 4 + 5 = 9.
When training TinyLLM on the 40 examples, I did not tell it what these tokens represented. Nor did I tell it that the relationship among the three tokens in each row was arithmetic addition. TinyLLM was simply trained to predict a particular next token when given a particular pair of input tokens. The training process was exactly as described above.
The resulting model not only remembered all 40 training examples but also demonstrated an ability to reason: remarkably, it produced the correct answers to the underlying addition problems without a single error. Saying that TinyLLM “grasped” the principle of arithmetic addition may seem inappropriate, because the model consists of nothing more than the cold, impersonal numbers shown in Figure 4. Can a collection of numbers, with no body heat, really “grasp” anything? The idea seems difficult to reconcile. Yet it did, in fact, answer all 55 addition problems correctly! Perhaps this is the very essence of artificial intelligence!
The Intelligence of Large AI Models:
The TinyLLM discussed here requires relatively little mathematics and only five or six hundred lines of VBA code. It is a simple, single-layer form of “deep learning.” Small as it is, it nevertheless possesses a certain degree of arithmetic intelligence because its training data come from the domain of arithmetic. This small mathematical model inhabits a very limited world, containing only ten tokens, with arithmetic addition as its underlying rule.
Very large AI models are different. GPT-3, for example, uses 96 layers of “deep learning.” Its vast training dataset consists of digitized language used in human society. The underlying rules include the grammatical structures of various languages, as well as arithmetic addition, since GPT-3 can also perform arithmetic. Consequently, the trained GPT-3 communicates in language we can understand and displays forms of logic familiar to us. In other words, the kind of data we feed into a model shapes the kind of AI it becomes.
Nevertheless, both the small mathematical TinyLLM discussed here and a large commercial model such as GPT-3 perform the same single task: predicting the next token. In TinyLLM, the next token is the answer. In GPT-3, however, many successive tokens are needed to form a sentence. Whether that sentence demonstrates intelligence requires further explanation. We should assess AI’s intelligence by its ability to handle situations it has never encountered, rather than by its memory alone. For example, suppose we give AI three sentences during training: “Person A puts the shoes into a red box.” “Person B puts the red box into a cabinet.” “Person C moves the cabinet into the next room.” After successful training, we ask: “Where are the shoes?” Notice that none of the training sentences explicitly states, “The shoes are in such-and-such a place.” The question therefore presents a new situation. If AI answers, “In the next room,” it demonstrates intelligence.
A simple explanation of how AI acquires this ability is that, during training, it works to “build a network of relationships. Then, through successive layers of mathematical operations that follow these connections, it links the relevant information, calculates the probabilities of the next token, and selects that token.”
Some may wonder whether such cold, mechanical calculations—predicting the next token—can truly qualify as intelligence. Does AI have consciousness? Professor Hinton, often called the godfather of AI, believes that AI possesses both intelligence and consciousness. For such questions, interested readers can ask large AI systems such as ChatGPT, Gemini, or Claude from Anthropic, or explore the many discussions on YouTube. Other questions—whether AI might destroy humanity and how humans can maintain control over it—are also the subject of intense debate.
Conclusion：
Although the TinyLLM discussed in this article is small, it possesses the core capabilities of AI: it can remember, reason, and generalize. It operates within a small world of arithmetic. By training it on examples embodying the principles of arithmetic, I turned it into an AI capable of performing addition. ChatGPT, by contrast, operates within the vast world of human society. By training it on the essential patterns of human language, OpenAI turned it into an AI capable of responding in ways that reflect human thought.
The intelligence displayed by such large AI models undeniably benefits from digital advantages that traditional carbon-based life can scarcely match. With computers’ immense, seemingly limitless memory and capacity to process vast quantities of data, these large language models integrate an ocean of knowledge, displaying an unparalleled breadth of perspective. Their astonishing speed of computation and development also places them far beyond what human intelligence can readily achieve. As AI’s surpassing of human capabilities becomes an inevitable historical tide, this invention of AI represents more than another technological leap: it brings human civilization to an unprecedented turning point. This is a revolution that redraws the boundaries of intelligence. Through this profound transition, human civilization will undergo sweeping transformation and reorganization. (October 3, 2026 — New Jersey)