Whenever someone asks me, “how does ChatGPT actually work?”, I typically encounter two ready-made answers: either “it’s magic” disguised as an explanation, or a deep dive into advanced linear algebra and calculus that loses the listener by the second sentence. It doesn’t have to be that way. It is entirely possible to explain it clearly—without simplifying to the point of falsehood, and without requiring a math degree. That is what I aim to do here.
It’s Not Magic: It’s a Massive Next-Word Predictor
At its core, an LLM (Large Language Model) does essentially one thing: given a piece of text, it calculates the most probable word to follow. It operates on the same fundamental principle as your smartphone keyboard’s autocomplete when it suggests the next word as you type—except it has been trained on a massive fraction of all publicly written text on the internet, with billions of fine-tuned internal parameters, rather than just your personal chat history.
The conversation that feels so fluid is, mechanically, this predictive calculation repeated hundreds of times: the model predicts a word, appends it to the end of the text, examines this new (slightly longer) sequence, and predicts the next word—again and again—until it determines that the response is complete. There is no conscious “thinking” happening between one word and the next; there is a finely calibrated statistical pattern engine executing at lightning speed. Generative AIs do not think like human beings; they simulate reasoning through statistical calculations and patterns learned from vast datasets. What appears to be “thought” is actually a highly sophisticated sequential prediction pipeline.
This does not diminish the achievement. Predicting the next word with sufficient accuracy to write functional code, summarize complex contracts, or explain subtle humor requires that the model’s internal numbers capture a remarkably rich representation of how language—and the world it describes—is structured. Understanding that the foundation is prediction rather than conscious reasoning explains nearly every counter-intuitive behavior these systems exhibit, which we will unpack below.
From Text to Numbers: Tokens
A model does not read text the way humans do. Before processing, text is divided into smaller chunks called tokens—think of them as pieces in a linguistic jigsaw puzzle. A token can be an entire word (“house”), a word fragment (“pre” + “dict” + “ion”), or even a single punctuation mark.
This tokenization stage explains some peculiar LLM quirks. For example, asking a model to count how many times the letter “R” appears in the word “strawberry” is a surprisingly tricky test. The model is not inspecting the letters S-T-R-A-W-B-E-R-R-Y individually; it is seeing one or two tokens representing the entire word as a unified chunk. It is akin to asking someone to count the individual letters in a word written in a script they only recognize by sight, without spelling it phonetically.
Giving Meaning to Numbers: Embeddings
Each token is converted into a list of numbers—an embedding—which maps the semantic meaning of that chunk into a high-dimensional mathematical space. Words with similar meanings or contexts end up positioned close to one another in this space. A famous illustrative example: if you take the coordinate vector for “king”, subtract the vector for “man”, and add the vector for “woman”, the result lands very close to the vector for “queen”. The model was never explicitly programmed with the rule “a queen is a female king”; that relationship emerged organically from how those words co-occur in human language.
It is this high-dimensional mathematical space, rather than a hardcoded rulebook or dictionary, that carries the model’s “understanding” of language.
The Central Breakthrough: Attention
A single word can change its meaning dramatically depending on its surrounding context. “I sat on the river bank” and “I deposited money at the bank” use the exact same word for entirely different concepts. A human reader resolves this ambiguity effortlessly by reading the rest of the sentence. LLMs achieve this through a mechanism called attention, the seminal 2017 technical breakthrough (the Transformer architecture) that made modern language models possible.
In practice, attention allows each token to “attend to” (look at) all other tokens in the passage and evaluate which ones are relevant to understanding its own meaning within that specific context. In the bank example, the token “river” steers the meaning of “bank” in one direction, while the token “money” pulls it toward another. Modern models execute this across multiple “attention heads” simultaneously: one head might track grammatical agreement (subject-verb concordance), another resolves pronouns (identifying who “he” refers to two sentences back), while another tracks the overarching topic. These attention layers are stacked dozens of times, refining understanding at each successive layer.
Input text
↓
Tokens
↓
Embeddings (numbers with semantic meaning)
↓
Multi-layer Attention (each word attends to all others)
↓
Probability distribution over all possible next words
↓
Chosen word, and the cycle repeats
How Does the Model Learn? Training
Training an LLM begins in an almost deceptively simple manner: take an enormous corpus of text (billions of web pages), mask the next word in each excerpt, and task the model (which starts with completely randomized numbers) with guessing what comes next. Initially, it gets almost everything wrong. Each mistake is evaluated against the correct ground truth, and an optimization algorithm subtly adjusts the model’s billions of internal numbers (its parameters) in the direction that would have made that guess slightly better. This process is repeated trillions of times.
At no point in this phase does a human engineer teach the model formal grammar rules. Grammar, world knowledge, rhetorical styles, and even rudimentary logical reasoning emerge indirectly as a natural byproduct of becoming progressively better at predicting the next word in text written by humans who understood grammar, facts, and logic.
From “Completing Sentences” to “Conversing Like an Assistant”: Fine-Tuning
A model trained solely on next-word prediction knows how to continue a text, but it does not inherently behave like a cooperative assistant. If asked a question, it might simply follow it with another related question rather than answering, because that is statistically common in conversational transcripts.
To resolve this, a second, more focused phase is applied: fine-tuning (instruction tuning). The model is trained on curated examples of dialogues between a user asking for assistance and a helpful, direct response. This is further refined through RLHF (Reinforcement Learning from Human Feedback), where humans evaluate multiple candidate answers to reward helpful, safe, and honest responses. This phase transforms a raw predictive text engine into an instruction-following assistant.
How the Response Is Formed: Generation and Temperature
When generating a response, the model executes the cycle described earlier: it calculates the probability distribution across all possible tokens for the next position, selects one, appends it to the text, and repeats. The selection is not always the single highest-probability token. A parameter called temperature controls how much the model is permitted to sample less obvious candidate tokens. A low temperature makes responses predictable and deterministic; a higher temperature introduces variety and creative divergence. This is why identical prompts can produce slightly different answers across sessions.
This process also operates within a hard constraint: the context window—the maximum number of tokens the model can process at any single instant. Anything beyond this threshold drops off the input; the model cannot reference it, not because it “forgot” in the human sense, but because it physically fell out of the active computational window.
Why Do They Hallucinate?
This is the most critical takeaway: the model does not possess a verified database of facts that it checks before responding. It produces the most statistically plausible sequence continuation—period. Most of the time this aligns with reality, because truthful statements are far more frequent and consistent across human literature than random falsehoods. But when the most linguistically plausible pattern does not match real-world facts—a convincing-sounding scientific citation, a nonexistent library function that “makes intuitive sense”, or a wrong date delivered with total authority—the model has no native internal fact-checking mechanism to detect it. It simply does not know what it does not know.
This is why web search augmentation (RAG) is so effective: it does not rewrite the underlying generation mechanics, but it injects verifiable, up-to-date source text directly into the context window, giving the model factual reference material to anchor its predictions.
What Does This Explain About AI Behavior?
Connecting the dots: operating on tokens explains why models struggle with character-counting tasks. Generating the most plausible continuation explains why they sound supremely confident even when incorrect. And being statistical pattern engines explains why they excel at tasks rich with human examples (writing, summarization, coding in popular languages) while faltering on rare edge cases or domains demanding absolute mathematical certainty.
There is one common misconception worth clarifying: when using ChatGPT or Claude with memory features enabled, and it recalls in a new session that you are a Software Engineer, it may feel as though the model retained that memory internally. It did not. The model itself remains entirely stateless between conversations; every new session starts from a blank context window. What actually happens is that the surrounding application product stores that metadata in a standard database and prepends a summary into the beginning of your new session’s context window. To the model, it is indistinguishable from you having re-typed “by the way, I am a Software Engineer”. The memory is genuine and useful, but it lives in the software wrapper, not in the statistical engine.
Understanding these mechanics transforms how I use these tools daily: I do not distrust them cynically, but nor do I trust an answer blindly simply because it is eloquently phrased. Despite rapid technological leaps, linguistic fluency and factual correctness remain completely decoupled in large language models—a reality worth remembering the next time an answer appears suspiciously fast and confident.