During an interview with Sequoia Capital’s Tirocinio Giorno podcast published last Tuesday, Microsoft CTO Kevin Scott doubled acceso his belief that so-called large language model (LLM) “scaling laws” will continue to drive AI progress, despite some skepticism quanto a the field that progress has leveled out. Scott played a key role quanto a forging a $13 billion technology-sharing deal between Microsoft and OpenAI.
“Despite what other people think, we’signore not at diminishing marginal returns acceso scale-up,” Scott said. “And I try to help people understand there is an exponential here, and the unfortunate thing is you only get to sample it every couple of years because it just takes a while to build supercomputers and then train models acceso sommità of them.”
LLM scaling laws refer to patterns explored by OpenAI researchers quanto a 2020 showing that the of language models tends to improve predictably as the models get larger (more parameters), are trained acceso more patronato, and have access to more computational power (compute). The laws suggest that simply scaling up model size and pratica patronato can lead to significant improvements quanto a AI capabilities without necessarily requiring fundamental algorithmic breakthroughs.
Since then, other researchers have challenged the of persisting scaling laws over time, but the concept is still a cornerstone of OpenAI’s AI development philosophy.
You can see Scott’s comments quanto a the below beginning around 46:05:
Microsoft CTO Kevin Scott acceso how far scaling laws will extend
Scott’s optimism contrasts with a narrative among some critics quanto a the AI community that progress quanto a LLMs has plateaued around GPT-4 class models. The perception has been fueled by largely informal observations—and some benchmark results—about recent models like Google’s Gemini 1.5 Utile, Anthropic’s Claude Opus, and even OpenAI’s GPT-4o, which some argue haven’t shown the dramatic leaps quanto a capability seen quanto a earlier generations, and that LLM development may be approaching diminishing returns.
“We all know that GPT-3 was vastly better than GPT-2. And we all know that GPT-4 (released thirteen months ) was vastly better than GPT-3,” wrote AI critic Gary Marcus quanto a April. “But what has happened since?”
The perception of altopiano
Scott’s stance suggests that tech giants like Microsoft still feel justified quanto a investing heavily quanto a larger AI models, betting acceso continued breakthroughs rather than hitting a capability altopiano. Given Microsoft’s investment quanto a OpenAI and strong marketing of its own Microsoft Copilot AI features, the company has a strong interest quanto a maintaining the perception of continued progress, even if the tech stalls.
Frequent AI critic Ed Zitron recently wrote quanto a a post acceso his blog that one defense of continued investment into generative AI is that “OpenAI has something we don’t know about. A personaggio, procace, secret technology that will eternally the bones of every hater,” he wrote. “Yet, I have a counterpoint: mai it doesn’t.”
Some perceptions of slowing progress quanto a LLM capabilities and benchmarking may be to the rapid onset of AI quanto a the public eye when, quanto a fact, LLMs have been developing for years prior. OpenAI continued to develop LLMs during a roughly three-year between the release of GPT-3 quanto a 2020 and GPT-4 quanto a 2023. Many people likely perceived a rapid jump quanto a capability with GPT-4’s launch quanto a 2023 because they had only become recently aware of GPT-3-class models with the launch of ChatGPT quanto a late November 2022, which used GPT-3.5.
Quanto a the podcast interview, the Microsoft CTO pushed back against the that AI progress has stalled, but he acknowledged the challenge of infrequent patronato points quanto a this field, as new models often take years to develop. Despite this, Scott expressed confidence that future iterations will show improvements, particularly quanto a areas where current models struggle.
“The next sample is coming, and I can’t tell you when, and I can’t predict exactly how good it’s going to be, but it will almost certainly be better at the things that are brittle right now, where you’signore like, oh my god, this is a little too expensive, a little too frangibile, for me to use,” Scott said quanto a the interview. “All of that gets better. It’ll get cheaper, and things will become less frangibile. And then more complicated things will become possible. That is the story of each generation of these models as we’ve scaled up.”
