How Large Language Models Actually

Work
Tokens, embeddings, attention, training stages, sampling, and context windows — an
explanation of the machinery beneath modern language models, without the mysticism.

Starting From The Right Question

A language model does one thing: given a sequence of text, it produces a probability distribution
over what comes next. That is the whole objective. Everything that looks like reasoning, translation,
summarization, or code generation is an emergent consequence of doing that one task extremely
well over an enormous quantity of text.

This framing is worth holding onto, because it explains both the capabilities and the failure modes.
A system optimized to produce plausible continuations will produce a plausible continuation even
when it has no grounds for one — which is precisely what a fabricated citation is. The behavior is
not a bug bolted onto an otherwise truthful system; it is the objective functioning as specified.

Tokens: The Units of Text

Models do not read characters or words. They read tokens, produced by a tokenizer that splits text
into subword pieces drawn from a fixed vocabulary, typically between 30,000 and 200,000 entries.

Most modern tokenizers use byte-pair encoding. Training starts with individual bytes and
repeatedly merges the most frequent adjacent pair into a new token. Common words end up as
single tokens; rarer words fragment. "The" is one token. "Tokenization" might be two or three. An
unusual surname or a chemical formula might be six.

Subword tokenization is a compromise between two bad extremes. Character-level vocabularies are
tiny but produce sequences so long that modeling long-range structure becomes expensive.
Word-level vocabularies produce short sequences but cannot represent anything unseen during
training. Subwords give a manageable vocabulary with no out-of-vocabulary failures, since worst
case a novel string decomposes into bytes.

Several practical quirks follow directly from tokenization. Character-level tasks are unnaturally
hard: asking a model to count the letters in a word requires reasoning about the interior of tokens it
perceives as atomic units, which is why such questions produce errors out of proportion to their
apparent difficulty. Arithmetic suffers similarly, because numbers split inconsistently — "1234"
may become "123" and "4", destroying place value. And languages underrepresented in tokenizer
training consume two to four times more tokens per unit of meaning, making them proportionally
more expensive to process.

Embeddings: Meaning as Geometry

1
How Large Language Models Actually Work

Each token identifier is mapped to a vector of a few thousand dimensions via a lookup table learned
during training. These embeddings place tokens in a space where geometric relationships encode
semantic ones. Tokens used in similar contexts land near each other, and directions in the space
acquire consistent meanings — the classic demonstration being that the offset from "king" to
"queen" resembles the offset from "man" to "woman."

Nothing about this is designed. It emerges because arranging vectors this way lets the model predict
text better, and gradient descent finds arrangements that lower loss.

Position must be supplied separately, because the attention mechanism at the model's core is
inherently order-agnostic — without positional information, "the dog bit the man" and "the man bit
the dog" would be indistinguishable. Early models added learned position vectors. Current models
predominantly use rotary position embeddings (RoPE), which rotate query and key vectors by an
angle proportional to position. This encodes relative distance in a way that generalizes better to
sequence lengths beyond those seen in training, and is a large part of why context windows have
grown so quickly.

Attention: The Central Mechanism

The transformer's key innovation is self-attention, which lets every token gather information from
every other token in the sequence, weighted by learned relevance.

Each token produces three vectors through learned linear projections: a query representing what it
is looking for, a key representing what it offers, and a value carrying the content it contributes.
Attention weight from one token to another is the dot product of the first's query with the second's
key, scaled by the square root of the dimension, then normalized across all positions with a
softmax. Each token's output is the weighted sum of all value vectors.

The intuition is a soft, differentiable lookup. When processing "it" in "The trophy did not fit in the
suitcase because it was too large," the query from "it" aligns more strongly with the key from
"trophy" than from "suitcase," so the representation of "it" absorbs information about the trophy.
Coreference resolution, syntactic dependency, and long-range factual association all emerge from
this mechanism without being programmed.

Multi-head attention runs many such operations in parallel with separate projections, then
concatenates the results. Different heads specialize — some track syntax, some track positional
patterns, some track entity relationships. Interpretability research has identified specific circuits,
such as "induction heads" that detect a repeated pattern and continue it, which appear to underlie
much of in-context learning.

In generative models, attention is causal: a mask prevents any position from attending to later
positions. Without it, the training task would be trivially solvable by looking ahead, and the model
would learn nothing useful about prediction.

2
How Large Language Models Actually Work

Attention's cost is quadratic in sequence length, since every token attends to every other. Doubling
context quadruples the computation. This is the fundamental constraint on context windows, and an
enormous amount of engineering — FlashAttention's memory-efficient kernels, grouped-query
attention, sliding-window and sparse patterns, various linear approximations — exists to soften it.

The Full Architecture

A transformer stacks dozens of identical blocks, each containing two sub-layers.

The first is multi-head attention, which mixes information between token positions. The second is a
position-wise feed-forward network — typically two linear layers with a nonlinearity, expanding to
roughly four times the model dimension and projecting back — which processes each position
independently. This is where most of the parameters live, and evidence suggests it functions
substantially as key-value memory storing factual associations learned during training.

Each sub-layer is wrapped in a residual connection and layer normalization. The residual
connection, adding a sub-layer's input to its output, is what makes hundred-layer networks trainable
at all: it provides a direct gradient path to early layers and lets each block learn an incremental
refinement rather than a complete transformation. It also gives rise to the "residual stream" framing
used in interpretability work, where the residual pathway acts as a shared communication channel
that each block reads from and writes to.

Many recent models replace the dense feed-forward layer with a mixture of experts: many parallel
expert networks with a small router that activates only two or so per token. Total parameter count
grows enormously while computation per token stays roughly constant, which is how models with
hundreds of billions of parameters remain economically servable.

After the final block, a linear projection maps the last position's vector to one logit per vocabulary
entry, and a softmax converts logits to probabilities.

Training in Three Stages

Pretraining is where capability originates. The model sees trillions of tokens of text — web pages,
books, code, academic papers — and is trained to predict each next token. The loss is cross-entropy
between predicted distribution and actual token. This is self-supervised: no human labels required,
since the text supplies its own targets. Learning to predict text well requires implicitly learning
grammar, factual associations, code semantics, translation, and a great deal of reasoning structure,
because all of those help lower prediction loss.

This stage dominates cost, consuming thousands of accelerators for weeks or months. Scaling laws
describe the tradeoff empirically: loss falls predictably as a power law in parameters, data, and
compute, and compute-optimal training requires scaling data alongside parameters rather than
simply building bigger models on fixed corpora.

3
How Large Language Models Actually Work

A pretrained model is not yet useful as an assistant. Asked a question, it may continue with more
questions, because that is a plausible continuation of text containing a question.

Supervised fine-tuning adapts it. Training continues on curated examples of instructions paired
with high-quality responses, teaching the format of helpful assistance. This stage is orders of
magnitude smaller than pretraining and teaches presentation rather than knowledge.

Preference-based alignment refines behavior further. In reinforcement learning from human

feedback, humans compare pairs of model outputs; a reward model learns to predict these
preferences; the language model is then optimized against that reward, usually with a penalty for
drifting too far from the fine-tuned starting point. Direct preference optimization achieves similar
ends without a separate reward model. Constitutional methods substitute a written set of principles
for much of the human labeling. This stage shapes tone, refusal behavior, calibration, and
instruction-following — most of what users experience as the model's character.

Generation and Sampling

The model outputs a probability distribution. Turning that into text requires a choice, and the choice
materially affects output quality.

Greedy decoding always takes the highest-probability token. It is deterministic and produces
repetitive, often degenerate text, because natural language is not the maximally probable
continuation at every step.

Temperature rescales logits before the softmax. Below 1.0 sharpens the distribution toward
confident predictions; above 1.0 flattens it toward diversity. At zero it becomes greedy.

Top-k restricts sampling to the k most likely tokens. Top-p (nucleus) sampling instead takes the
smallest set whose cumulative probability exceeds p, adapting the candidate pool to the model's
confidence — narrow when the next token is nearly certain, wide when genuinely open. Top-p is
generally preferred for this reason.

Practical guidance: low temperature with low top-p for factual extraction, code, and structured
output; moderate to high for creative writing. Repetition penalties discourage loops but distort
distributions if set aggressively.

Generation is autoregressive: each token is appended and fed back in. This makes inference
inherently sequential and latency-bound. The KV cache avoids recomputing keys and values for all
previous tokens at each step, trading memory for speed, and its size — growing with both context
length and batch size — is often the binding constraint on serving throughput. Speculative
decoding, where a small draft model proposes several tokens that the large model verifies in one
pass, is a common further optimization.

Context Windows and Their Limits

4
How Large Language Models Actually Work

The context window is the maximum tokens the model can attend to: the system prompt,
conversation history, retrieved documents, and generated output all share this budget. Windows
have grown from a couple thousand tokens to hundreds of thousands.

Two things are commonly misunderstood. First, the model is stateless between requests. It has no
memory of prior conversations; the appearance of memory comes from resending history each time.
Second, a long window does not guarantee uniform attention across it. Evaluations consistently find
a "lost in the middle" effect, where information at the beginning and end of a long context is used
far more reliably than material buried in the middle. Placing critical instructions at the edges is a
real and measurable practice, not superstition.

Retrieval-augmented generation exists partly because of these limits and partly for grounding.
Documents are chunked, embedded, and stored in a vector index. At query time, semantically
similar chunks are retrieved and inserted into the prompt. This grounds output in specified sources,
allows updating knowledge without retraining, and enables citation. Its quality depends far more on
retrieval quality than on the model — bad chunking and poor ranking produce confident answers
built from irrelevant context.

Why Models Get Things Wrong

Several failure modes follow directly from the mechanics above rather than from implementation
defects.

Fabrication arises because the objective rewards plausible continuations, and a plausible-looking
citation is easy to generate while verifying one requires an external source. Retrieval and tool use
address this by supplying grounds; the model alone has only its learned distribution.

Calibration gaps occur because expressed confidence is a linguistic style learned from training
text, not a readout of internal uncertainty. Alignment training improves the correlation but does not
make it reliable.

Knowledge cutoffs are inherent: parameters encode the training corpus and nothing after it. Only
tools or retrieval bridge that gap.

Prompt sensitivity persists because the model conditions on the exact token sequence it receives.
Reformulating a question changes the conditioning and can change the answer, which is
unsatisfying but expected from a conditional distribution.

Sycophancy — deferring to a confidently stated but incorrect user claim — is partly a consequence
of preference training, since human raters tend to prefer agreeable responses.

Adapting a Model Without Retraining It

Full fine-tuning updates every parameter, which requires memory for the weights, their gradients,
and optimizer state — several times the model's size — and produces a complete new copy per task.
For most purposes this is unnecessary.

5
How Large Language Models Actually Work

Parameter-efficient fine-tuning exploits the observation that task adaptation appears to occupy a
low-dimensional subspace. Low-rank adaptation (LoRA) freezes the original weights and learns a
pair of small matrices whose product is added to selected weight matrices. If the original is a
4096-by-4096 matrix, a rank-16 adapter trains roughly 130,000 parameters instead of 16.7 million.
Quality on the target task is typically close to full fine-tuning, the trained adapter is a few
megabytes rather than hundreds of gigabytes, and many adapters can be swapped against one base
model in memory. Quantized variants push the base model to four-bit precision during training,
bringing fine-tuning of substantial models within reach of a single accelerator.

The strategic question is when to fine-tune at all. Fine-tuning teaches form efficiently — a specific
output format, a domain register, a classification boundary, a consistent style. It is a poor
mechanism for teaching facts, which change and which the model may still fabricate around. For
factual grounding, retrieval is both cheaper and more maintainable. A reasonable escalation order
is: improve the prompt, add few-shot examples, add retrieval, then fine-tune — because each step
costs more to build and maintain than the last, and the earlier steps solve most problems.

Quantization reduces numerical precision at inference. Weights trained in 16-bit float can often
run at 8-bit or 4-bit with modest quality loss, cutting memory roughly proportionally and improving
throughput on memory-bandwidth-bound workloads. Because generation is dominated by moving
weights from memory rather than arithmetic, quantization frequently produces near-linear
speedups. Distillation is the complementary approach: train a small model to imitate a large one's
output distribution, yielding a model that is cheap to serve and specialized to the distilled behavior.

Tool Use and Agentic Loops

A model that only emits text is limited to what its parameters encode. Tool use removes that ceiling.
The mechanism is straightforward: the prompt describes available functions with their parameter
schemas, and the model — trained to produce structured output in this format — emits a call. The
surrounding application executes it, appends the result to the context, and the model continues with
real data in hand.

This is what makes calculators, database queries, code execution, and web search available to a
language model, and it directly addresses several of the failure modes described earlier. Arithmetic
stops depending on tokenization artifacts. Knowledge cutoffs stop mattering for anything
searchable. Claims become verifiable because a source is retrieved rather than recalled.

Chaining these calls produces an agent: a loop in which the model plans, acts, observes results, and
revises. The practical difficulties are less about the model than about the loop. Errors compound
across steps, so a workflow of ten actions with 95% per-step reliability succeeds only about 60% of
the time. Context accumulates, and long tool outputs crowd out the original instructions. And an
agent given real write access can take real destructive actions, which is why permission boundaries,
dry-run modes, and human confirmation on consequential operations are engineering requirements
rather than niceties.

6
How Large Language Models Actually Work

Evaluation
Benchmark scores are the least informative available signal about whether a model will work for a
specific task. Public benchmarks leak into training corpora, which inflates results without
corresponding capability. They measure aggregate performance on academic tasks that rarely
resemble production workloads. And small differences in reported numbers frequently fall inside
the noise of prompt formatting choices.

What works instead is building an evaluation set from your own data. Collect a few hundred real
inputs with known-good outputs, including the edge cases that have caused problems. Grade
automatically where the answer is checkable — exact match, schema validity, whether generated
code passes tests — and use held-out human review or a separate model as judge for open-ended
quality, spot-checking the judge against human labels to confirm it agrees.

Run this suite on every prompt change, every model version, and every retrieval configuration
change. Without it, prompt engineering is unfalsifiable: every change feels like an improvement
because you tested it on the three examples that motivated it. Teams that build evaluation
infrastructure early consistently ship more reliable systems than teams that rely on impressions, and
the gap widens as systems grow more complex.

The Economics of Serving

Cost and latency are both functions of tokens, and understanding the asymmetry between the two
phases of inference is useful.

The prefill phase processes the entire input prompt in parallel and is compute-bound. The decode
phase generates output tokens one at a time and is memory-bandwidth-bound, since each token
requires reading the full weight set. This is why output tokens are typically priced several times
higher than input tokens, and why generating a long response takes disproportionately longer than
reading a long prompt.

Practical consequences follow. Time to first token is dominated by prompt length; total latency is
dominated by output length, so instructing a model to be concise measurably improves both cost
and responsiveness. Batching requests improves throughput substantially during decode because
weights are read once for the whole batch, though it raises individual latency. And prompt caching
— reusing computed keys and values for an unchanged prefix such as a long system prompt or
document — can reduce both cost and latency dramatically for repeated calls sharing context,
which is why stable content belongs at the beginning of a prompt and variable content at the end.

What This Buys You

Knowing the mechanics changes how you use these systems. You understand why examples in a
prompt help so much: in-context learning is pattern continuation, and demonstrations are the
pattern. You understand why chain-of-thought prompting improves reasoning: each generated
token is additional computation conditioned on prior tokens, so producing intermediate steps gives

7
How Large Language Models Actually Work

the model more forward passes to work with. You understand why placing instructions at context
edges matters, why temperature should differ between code and prose, why token counts drive both
cost and latency, and why grounding through retrieval is not optional for factual work.

The machinery is comprehensible. Attention, residual streams, next-token prediction, and sampling
account for a remarkable amount of observed behavior, and treating these systems as
understandable engineering artifacts produces better results than treating them as oracles or as
magic.