Image by Author

Training a large language model requires adjusting billions of parameters to make the model’s predictions better. The mechanism behind those adjustments is gradient descent, a method for walking downhill on a mathematical landscape where the height at any point represents how wrong the model currently is.

But there’s a problem. Computing the exact direction of “downhill” requires processing the entire training dataset at once. For modern language models trained on hundreds of billions of words, that’s simply not feasible. The solution is stochastic gradient descent, which estimates the direction of improvement from a small random sample of the data rather than computing it exactly from all of it.

This article explains what gradient descent is, why the exact version is impractical, how the stochastic version solves that problem, and what statistical principle makes it work despite being intentionally imprecise.

What Gradient Descent Is

Start with a simpler picture. Imagine you’re standing on a hillside in dense fog, and your goal is to reach the lowest point in the valley. You can’t see the valley, but you can feel the slope of the ground beneath your feet. At any point on the hill, the slope tells you which direction is downward. Take a step in that direction, feel the slope again, repeat. Eventually you reach the bottom.

Gradient descent works the same way, but the “hill” is the loss surface, a mathematical object where every possible setting of the model’s parameters corresponds to a point, and the height at each point is the training loss for those parameters. The gradient is the slope: a vector pointing in the direction of steepest increase in loss. Moving in the opposite direction of the gradient means moving toward lower loss.

The update rule is simple in concept. At each step, compute the gradient of the loss with respect to every parameter, then nudge each parameter a small amount in the direction that reduces loss. The size of each nudge is controlled by the learning rate, a number chosen by the practitioner. Repeat this process many thousands or millions of times, and the model gradually improves.

The Problem: The True Gradient Is Expensive

Computing the exact gradient requires evaluating the loss across the entire training dataset. For a dataset with one billion examples, that means a billion forward passes through the model before you can take even a single update step.

This is computationally prohibitive. By the time you’d computed one exact gradient step, a model using estimates could have taken thousands of update steps and made substantial progress. Waiting for exactness defeats the purpose when approximation gets you there faster.

There’s also a subtler statistical point. Using the entire dataset for every gradient gives you precision that may not actually be needed. If your training data contains many near-identical examples, computing the gradient over all of them is partly redundant. The signal from any large, representative sample of the data is close to the signal from the full dataset.

The Fix: A Minibatch Is a Random Sample

Stochastic gradient descent replaces the full dataset with a minibatch, a small, randomly drawn subset of training examples, typically containing somewhere between 32 and 512 examples depending on the task and hardware constraints.

The gradient computed from a minibatch is not the true gradient. It’s a noisy estimate of it. But here’s the key insight: it’s an unbiased estimate. If you drew many different minibatches and computed the gradient from each, the average of those gradients would converge to the true gradient. No single minibatch gives you the exact direction, but each one gives you a reasonable approximation, and they don’t systematically point in the wrong direction.

This is a direct application of the law of large numbers. The expected value of the minibatch gradient equals the full-data gradient. With each update step, you may be moving in a slightly wrong direction, but on average you’re moving in the right one. Across thousands of steps, those small errors average out and the model converges.

This should feel familiar if you’ve thought about sampling and estimation in other statistical contexts. A poll of 1,000 people drawn at random from a population of millions gives you a noisy but unbiased estimate of the population’s views. A minibatch of 256 training examples gives you a noisy but unbiased estimate of the training gradient. The same logic applies.

The Speed-Noise Tradeoff

The practical consequence of this design is a tradeoff between speed and variance in the gradient estimate.

A larger minibatch produces a more accurate gradient estimate but requires more computation per step. A smaller minibatch produces a noisier estimate but lets you take more update steps in the same amount of time. Both extremes have practical costs.

With a very small minibatch (even a single example, the original “stochastic” version of the algorithm), the gradient estimate is noisy enough that individual steps may point in counterproductive directions. Progress is erratic. The loss doesn’t fall smoothly; it bounces around. With a very large minibatch approaching the full dataset, gradient estimates become precise but you take fewer steps for the same compute budget, and there’s some evidence that very large batches can make it harder for the model to find good solutions.

In practice, minibatch sizes in the hundreds have become a sensible default. The training loss doesn’t fall as smoothly as it would with exact gradient descent, but the trajectory trends downward, and the speed advantage is large.

Learning Rate: The Size of Each Step

The learning rate controls how far the model moves in the estimated gradient direction at each update step. Too large, and individual steps overshoot good parameter settings, causing the training loss to oscillate or diverge. Too small, and training progresses unnecessarily slowly.

Because minibatch gradient estimates are noisy, the learning rate interacts with batch size in a meaningful way. A larger batch size (less noisy gradient) can support a somewhat larger learning rate. A smaller batch size typically requires a more conservative learning rate to compensate for the higher variance in each step’s direction.

Practitioners often decrease the learning rate over the course of training, a practice called learning rate scheduling. The intuition is straightforward: early in training, large steps help the model find roughly the right region of parameter space quickly; later, smaller steps allow finer-grained convergence without overshooting.

Conclusion

Stochastic gradient descent makes large-scale model training feasible by replacing the exact gradient, which would require processing the entire dataset at every step, with an estimate computed from a small random sample. That estimate is noisy but unbiased, meaning individual steps may be imprecise while the overall trajectory trends in the right direction. The result is a training algorithm grounded in the same statistical logic as sampling and estimation: you don’t need to see the whole population to get a reliable read on it, as long as your sample is drawn randomly and repeated enough times.