Image by Editor

Large language models (LLMs) operate by building their text responses word by word, predicting at each step the next, “most likely” word to generate. In this output generation process, not only does response relevance matter, but also other criteria are at stake, such as coherence and creativity. Successfully capturing each of these properties to a certain extent entails mingling with the underlying word probability distributions that govern this process.

This article introduces the dynamics of LLMs’ output generation from a statistical viewpoint. We will examine how raw model scores are passed through several statistical filters and levers — like sampling thresholds, temperature, and penalty, to name a few — and how they can shape the ultimate process of selecting the next word to generate.

Although we will focus on exploring the final stages of LLMs’ internal process, if you are interested in an overview of the entire journey made by tokens across the models’ core architecture, the transformer, you can check out this related article.

Note: while we use the term ‘word‘ in the context of next-word prediction in LLMs for the sake of ease in discussion, in accurate technical terms, LLMs operate with tokens rather than words, i.e., linguistic units that can represent entire words, but also certain individual characters or common sub-words (like prefixes and suffixes).

The Foundation: Logits and Probability Distributions

Neural network architectures like the LLMs’ transformer can be designed to generate at their final layer of neurons some raw, unnormalized scores known as logits. Every possible word in a vocabulary (e.g. a target generation language like English) is assigned a logit score as a result of some in-depth processing.

Example: predicting the next word after “I really want to” might yield logits of 11.5 for “travel”, 6.7 for “eat”, and -2.1 for “run”. Interpreting raw logits is not straightforward; hence, a Softmax function transforms them into a standard, normalized probability distribution so that all values sum to 1. And so, a full statistical map of the vocabulary is built.

Shaping the Probability Distribution

It may seem like once we have the word probability distribution, the model always chooses the most probable one. However, LLMs add a bit of much-needed sophistication by using some parameters and strategies to reshape the “statistical curve” before conducting a word sampling from the distribution. Here are some of them:

• Temperature: it scales probabilities, such that a higher temperature flattens the distribution and makes it more uniform-like, which in practice enhances variance, creativity, and uncertainty in choosing the next word. Meanwhile, a low temperature sharpens the distribution, which heavily favors the most probable word(s) and turns the model behavior more deterministic.
• Top-k and top-p: these two are truncation strategies, necessary in real-world applications where the vocabulary is too large. Top-k considers only the top k most probable items as candidates for sampling the next word, whereas top-p (also called nucleus sampling) builds a pool of candidate words in decreasing order of probability, until the cumulative probability exceeds a threshold p. In both cases, the tail with less likely words is trimmed.

These common strategies are illustrated below:

Besides, some additional mechanisms can be implemented to reflect additional generation criteria or analyze the overall generated sequence more broadly:

• Frequency and presence penalties: this can be applied on top of the previous strategies to penalize the probability of words that have been previously generated, to avoid repetition and foster broader vocabulary in responses.
• Beam search: a look-ahead algorithm that explores several probable sequence paths in parallel to come up with the most probable sequence overall. It eliminates the greediness of immediate next-word sampling.

The concepts and mechanisms explained are typically combined to form a sequential pipeline. The logical steps in a simple pipeline may look like:

1. The model generates raw logits at its last neural layer.
2. Temperature is used to scale logits, after which the Softmax function converts scaled logits to percentages or word probabilities.
3. Top-p or top-k are used to truncate the distribution and keep only the most probable words as candidates for sampling.

Closing Remarks

Tuning the engine behind LLMs requires carefully balancing several criteria, like predictability, creativity, and avoidance of repetition, to optimize the process of generating outputs. This article took a tour of the statistical notions adopted in the process of choosing the next word in a sequence of text produced by these models, helping you gain an understanding of how models like Gemini, Claude, and ChatGPT make decisions on every single word they print on your screen when you are interacting with them, and why.