Image by Editor
Despite their outward complexity and seemingly “magical” capabilities, modern AI systems, such as language models and other generative models, are built on foundations that date back to the dawn of machine learning, or even earlier. In this article, we examine four of these foundations and explain how they apply to the latest AI systems.
From Slopes and Hyperplanes to Billions of Parameters
The goal of the most basic regression approach, linear regression, is to find the line function, or hyperplane in a broader sense, that best fits a set of training examples with known output labels $y$. This best fit is learned by adjusting the parameters of the associated mathematical function — the weights associated with input variables, or features — to map inputs to outputs as accurately as possible. Based on this same idea, modern AI systems built on deep neural network architectures use weights and biases. The core principle remains the same: fitting the model’s weights to a set of labeled training data and determining which input features are most relevant for making a prediction. The massive difference is scale and speed: large language models (LLMs), for instance, manage up to billions of these weights across immense neural networks.
Softmax: Probabilities Turned Into Decisions
The Softmax function has been used in traditional classification models for decades. It turns raw scores, which indicate the extent to which an input example may belong to different classes, into interpretable probabilities. While these uses often boil down to categorizing simple data, such as customers, animals, and plant specimens, the concept remains at the heart of a critical part of LLMs: next-word generation. Eligible words in the vocabulary used for language generation play the role of “classes,” and the model uses a Softmax layer at its final stage to compute a class probability distribution across that vocabulary. So, the next time you see ChatGPT words gradually printed on screen, each one is a statistically probable candidate sampled from that distribution.
Learning From Mistakes: Gradient Descent
No machine learning model starts smart — not even the “smartest” ones. Models learn relationships and patterns by being exposed to many input-output examples, or training data. This is a trial-and-error process, and optimization algorithms such as Gradient Descent guide it by using derivative calculus to identify and gradually reduce prediction errors by nudging model weights. These guiding algorithms are still used in modern models, for instance, by calculating “how wrong an answer was” compared to a ground-truth label, or loss, and adjusting millions or billions of weights accordingly to reduce that error, slowly but steadily.
Overfitting and Regularization: Learn, Don’t Memorize
Overfitting is a common, classical machine learning problem. It happens when a model memorizes the training examples too closely, so that once trained, it struggles to handle new, real-world data properly. The problem tends to emerge as models increase in complexity, so it is no surprise that modern AI systems need to be carefully designed to prevent or address it. Among the existing solutions to overfitting is regularization, which restricts the model’s freedom during training and keeps it from getting too “attached” to certain patterns. In practice, by penalizing overly complex solutions through constraints, a system such as generative AI for image creation can remain flexible and somewhat creative instead of becoming a copy-paste machine.
How Other Classic Machine Learning Mechanisms Translated Into Modern AI
Other mechanisms and issues found in production machine learning systems have also shifted into cutting-edge AI system contexts:
- Train/test leakage in benchmarks: Similar to classical models that can give a false impression of excelling when test data slips into the training set, LLMs can inadvertently ingest benchmark prompts during their internet-scale pre-training stage, giving a false impression of astounding reasoning capability with little genuine intelligence underneath.
- Bias-variance trade-offs in human feedback loops and fine-tuning: Finding the right balance between underfitting and overfitting remains crucial today. In the context of using reinforcement learning from human feedback (RLHF) to align an AI system with human preferences, pushing the process too far and making this alignment too strict may cause the model to overfit to specific human preferences, degrading its creativity, flexibility, and breadth of knowledge.
- Cross-validation thinking for LLM evaluation: The multifaceted strategy of cross-validation for robust machine learning model evaluation is still applied to LLMs, for instance, to assess model performance across diverse prompts and open-source benchmark frameworks while seeking consistency in real-world performance.
Closing Remarks
We have outlined several examples of how classical machine learning concepts and strategies typically used in regression and classification are still crucial elements of the latest AI systems we use in daily life, including language models, generative AI, and more.
Looking even further into the era of agentic AI, some of these foundations are becoming even more important. An agent planning a complex task or navigating a digital environment requires the ability to make autonomous decisions. This ability is also rooted in the probabilistic and optimization concepts we just explored. In sum, these principles are still far from obsolete, and they have earned the right to be considered legacy concepts in the AI landscape.
