Image by Editor | Midjourney

Large language models (LLMs) like Claude, Gemini, and ChatGPT are normally non-deterministic: as part of their next-word generation process to build an output sequence, they use sampling strategies to add variability in their outputs. Put another way, this is where much of LLMs’ “sense of creativity” when answering your prompts comes from: if you repeatedly ask ChatGPT for a Japanese Haiku poem, you’ll likely get different ones, rather than always the same poem.

Here’s the catch: randomness in text generation can be controlled to some extent, by playing with some LLM sampling hyperparameters.

This article introduces and dives into the mechanisms behind two of them: temperature and top-K sampling.

Temperature

Temperature refers to the “certainty” of predictions: recall that an LLM-generated output is the result of solving a sequence of next-word prediction problems. In mathematical terms, temperature influences the probabilities of the model outputs, such that a lower temperature makes the model adopt a more confident, deterministic-like behavior, while a higher temperature makes its predictive behavior more random.

Temperature, denoted by T, normally takes values in [0, 1]. When T is closer to 1, next-word probabilities are distributed more evenly, introducing the likelihood that less “expected” words might be chosen. Meanwhile, when T is closer to 0, the model favors the highest-probability words, which narrows down options for the next word to predict.

Suppose for instance an LLM that is asked to generate a fairy tale. If the temperature is set as T = 0.9, a prompt like “Once upon a time, a king” might generate an overly creative (and perhaps even weird!) text to continue the story, such as “explored an underwater ship wreckage,” whereas if T = 0.2, then the follow-up generate text might sound much more predictable, like: “ruled a prosperous kingdom“.

This visual example illustrates how T affects the sharpness or homogeneity of next-word probability distributions:

Next-word probabilities under different temperatures

In practice, moderate temperature values like 0.5 to 0.7 are typically used to balance creativity and coherence in responses.

A brief note on how temperature affects the innards of an LLM’s transformer architecture: while the softmax activation function responsible for computing raw next-word probabilities (logits) remains unaltered, what sampling hyperparameters like T do is adjusting these raw logits to yield a final next-word probability distribution that better reflects the specified sampling behavior.

Top-K

The purpose of this sampling hyperparameter is to restrict the available choices for the next word to predict, limiting it to the K most probable options only, leaving out the rest. This approach helps prevent generating irrelevant or incoherent outputs as a result of sampling low-probability tokens. Consequently, while top-K does not address a typical LLM issue like hallucinations, it partly helps reduce its chances of occurrence by discarding less likely words from the sampling process that LLMs conduct to decide on the next word to generate.

How Top-K affects raw next-word probabilities

Intuitively, the lower Top-K, the more restrictive and less creative the model’s decision-making behavior. For instance, in a chatbot responding to “How are you?”, it might limit responses to options like “I’m good,” “I’m fine,” or “I’m okay,” whereas if Top-K = 30, there is a broader range of options, leading to more diverse or creative responses like “I’m a bit tired but happy,” or “I’m feeling motivated today!”

Summary Guidelines

In summary, temperature and top-K can be combined to adjust the LLM behavior, balancing diversity and coherence in response:

• Low T and low Top-K produce highly focused and reliable responses, ideal for business or learning use cases
• Moderate T and higher Top-K yield more creative outputs, suitable for use cases like poem generation