Language models (or LMs for short) are highly sophisticated deep learning solutions trained upon vast volumes of text to learn to understand and generate human-like text in a variety of use cases. The are predominantly based on a revolutionary architecture called the Transformer, which can be designed to specialize in language understanding, language generation, or a balance between both.

With the variety of language tasks and use cases LMs can address — including question-answering, translation, text summarization, and text generation, to name a few — it is not surprising that traditional metrics to measure model performance, like accuracy and error metrics, are not enough to comprehensively capture the nuanced performance aspects of language understanding and generation, including semantic coherence and contextual appropriateness. This article unveils a popular LM metric to assess the quality of generated text: perplexity.

Perplexity: What it is, How to Calculate it, and How to Interpret it

Perplexity is a statistical measure frequently used to evaluate the performance of LM tasks involving text-generated outputs. LMs generate output text sequences word by word, repeatedly addressing a so-called next-word prediction problem in the process. Accordingly, perplexity quantifies how well the model predicts a sequence of words. Concretely, it reflects the model’s confidence in its predictions: a lower perplexity indicates that the model assigned higher probabilities to the chosen sequence of generated words, which means better performance.

The formula for calculating perplexity is:

Perplexity(W) = exp(-1/N ∑ log₂ P(w_i|w₁, …, w_i-1))

where W = (w₁, w₂, …, wₙ) is the sequence of words or tokens generated by the LM, and N is the sequence length. P(wᵢ|w₁,…,wᵢ₋₁) is the conditional probability of w_i being the next word to generate, given the previous i-1 words. The base-2 logarithm is used so that the formula resembles information theory measurements like entropy and the multiplication of many successive probabilities does not yield extremely low values. Lastly, the exponential function helps make the final result more interpretable.

And speaking of interpretability, let’s see how to interpret this measure. Its minimum possible value is 1, which occurs when the model predicts the data (expected words to generate) perfectly: this “perfect scenario” happens when all chosen words in the generated sequence have maximum probability. Conversely, higher perplexity values indicate that the LM experienced more uncertainty in deciding on the words to sequentially generate next.

Suppose the performance of three models is compared for generating a sequence of three words, with the following next-word probabilities for each generated word per model:

Model A: [0.7, 0.2, 0.1]
Model B: [0.6, 0.25, 0.15]
Model C: [0.5, 0.3, 0.2]

Intuitively, model A is the most self-confident in words generated overall, hence its perplexity is the lowest.

Using perplexity effectively entails having evaluation datasets or ground truth that represent the intended use case for the LM. Perplexity is not used to assess a model in isolation but rather to compare models trained on the same dataset. It should also be interpreted in the context of the task and domain. For example, general-purpose conversational models may have higher perplexities due to the variability of human dialogue than domain-specific fine-tuned LMs.

Limitations of Perplexity

Some limitations of this statistical measure are:

• Data Sensitivity: reliance on perplexity depends heavily on the quality and diversity of the evaluation dataset
• Comparison Across Models: perplexity values are not always comparable across different LM architectures
• Task-Specific Limitations: perplexity focuses solely on next-word prediction, being less suitable for more advanced tasks like question-answering or text summarization

Strategies to address these issues include complementing perplexity with other task-specific metrics like BLEU and ROUGE scores, using widely accepted datasets for benchmarking, and incorporating human judgment to assess output quality.