An LLM never sees your words.
It sees tokens.

A token is the smallest unit of text the model works with. It is usually not a whole word and not a single character — it sits in between, at the level of frequently-seen chunks. Common words like the are one token. Rare or long words get split: tokenization might become token + ization. Whitespace, punctuation, and even emoji all become tokens too.

Subwords are a compromise: small enough that a fixed vocabulary (typically ~50,000 to ~200,000 tokens) can express any text, large enough that common text stays short. That vocabulary size is a hard design constant of the model.

The vocabulary isn't hand-written. An algorithm called Byte-Pair Encoding (BPE) starts from raw characters and repeatedly merges the most frequent adjacent pair into a new single unit. Do that thousands of times over a huge corpus and the merges that survive are the vocabulary. Step through a tiny run below.

In the real training run this happens across billions of words, so the pairs that win are the ones that recur everywhere — ing, tion, the. That's why the tokenizer feels like it "knows" English morphology: it doesn't, it just kept the merges that paid off statistically.

Once the vocabulary exists, each token maps to a fixed integer ID — its row number in the vocabulary. The model never manipulates letters; it manipulates these IDs. Tokenizing is just a dictionary lookup from text chunk to number, and detokenizing is the same lookup in reverse.

That last line is why models are shaky at arithmetic: a number like 429 isn't a quantity to them, it's an arbitrary sequence of digit-fragment IDs. The model has to learn that 4+29 means four hundred twenty-nine — nothing in the IDs tells it so.

An integer ID carries no meaning on its own (ID 5,000 isn't "bigger" than ID 12). So the first thing the model does is look up each ID in an embedding table and replace it with a long list of numbers — a vector, often 2,000–12,000 dimensions wide. These vectors are learned, and tokens that behave similarly end up pointing in similar directions.

The sequence of token vectors (plus a position signal so order isn't lost) flows through many transformer layers. Attention lets every token look at every earlier token and mix in relevant context, so the vector for bank ends up different in "river bank" vs "bank account." After the final layer, the model produces one score — a logit — for every token in the vocabulary: "how likely are you to come next?"

Raw logits are unbounded scores. Softmax squashes the whole set into probabilities that sum to 100%. Then temperature reshapes that distribution before a token is drawn: low temperature sharpens it toward the top choice (deterministic, safe), high temperature flattens it (diverse, riskier). Drag the slider.

At temperature → 0 the model always takes the single highest-probability token (greedy) — repeatable, but flat. Turn it up and rarer tokens get a real chance, which is where both creativity and hallucination come from. This one knob is the difference between a boring answer and a wild one.

A model doesn't emit a sentence in one shot. It predicts one token, appends it to the input, and runs the whole thing again to predict the next — over and over until it emits a special stop token. Everything it has already written becomes part of the context for what comes next.

This is also why latency scales with output length: every single output token is a full forward pass through the network. Longer answers literally cost more compute, token for token.

Tokens aren't trivia — they're the unit of nearly every real constraint you hit when building with LLMs.