Image by Editor | Midjourney
Language models have powered many applications, including text generation, translation, question-answering, and more, and have themselves to be assets to many businesses.
In the center of the language model is the concept of the probability distribution. It’s what dictates how the model could predict words or sequences based on the input.
In this article, we will learn how the probability distribution works in typical language models.
Language Model Probability Distribution
First, let’s refresh our knowledge about probability a bit. Take, for example, the following text:
“The quick brown fox jumps over the lazy dog. The fox is clever and quick.”
The probability for each word using random sampling with replacement is shown in the table below.
Word frequency probabilities for the given sentence.
If the text above is the only corpus that our language model trained upon and we use only a sampling method to select the text, the above table will become the probability distribution for our model.
Probability distribution in statistics is the assignment of likelihood to all possible outcomes of a random variable. If discussing language models, it will be the likelihood of words where the probability distribution will quantify the likelihood of each word appearing given the input.
For example, given the input “The cat is,” the language model will assign the likelihood of the next words such as “sweet,” “cute,” “nice,” and many others. The model will try to select or sample the next words based on the probability distribution and the set rules.
Depending on the structure of the language models, the internal probability distributions will differ.
Let’s take the example of the traditional statistical language model n-gram models. It’s a simple statistical model where the probability distribution of a word is based on the preceding n-1 words.
If we make the n-gram model probability distribution formula, it will resemble the following:
In statistical terms, the independence assumptions above are called Markov assumptions. Given the set of n-1 words, what are the probabilities of the next word?
For example, in the trigram model (n=3), the probability of the next word will only depend on the immediately preceding word and the previous two words. The formula will follow the chain rule of probabilities, as shown below.
While the n-gram model has an explicit probability distribution, there are language models that have much more implicit distributions. This means that probability calculations are embedded within the architectures instead of directly visible calculations.
Transformer-based models, such as GPT, calculate the next word probability based on a combination of embeddings, attention mechanisms, and softmax normalization. This allows the model to predict probabilities over an entire vocabulary effectively.
There are even language models without probability distributions, which use a deterministic model such as one that is rule-based.
That’s all for now. I hope you understand the language model probability distribution better.
