Multiple reports have emerged in the last week stating that conventional “upscaling” – that is, the use of increasingly large datasets in an effort to produce better AI – is losing its effectiveness, and that concern has begun to mount within the AI industry. OpenAI insiders report that GPT-4’s replacement, codenamed “Orion,” is not reliably performing better than GPT-4 itself, as prominent AI researchers warn that up-scaling is reaching its limit.

This week, I’m going to be examining this particular story in greater depth, as well as the history of the AI field.

Bottom Line: Evidence is mounting that the current era of explosive, exponential AI growth may be coming to an end, as up-scaling of datasets loses effectiveness as a means to produce better AI. Though this has yet to be reflected on Wall Street, this news may bring a potential “AI bubble” closer to deflation.

 

Analysis: Growing Signs of AI Development Slowdown

Several weeks ago, Reuters reported that top AI scientists, as well as sources within OpenAI, have stated that the previously-rapid progression of AI development is beginning to hit serious structural roadblocks. Subsequently, other industry sources have made similar reports. Tech outlet The Information reported that, per OpenAI employees, the company’s next-gen successor to the now two-year-old GPT-4, codenamed Orion, is not reliably performing better than its predecessor at certain tasks, and that the rate of model improvement has decreased from previous iterations. Bloomberg has additionally reported similar slowdowns are occurring with the next iteration of Google’s “Gemini” and Anthropic’s “Clyde” AI models, suggesting that whatever is happening is not unique to any one LLM. Concerns have begun to spread within the industry, and the tech world more broadly, that a significant slowdown, or even bubble-burst, may be in the cards.

Tech Execs Remain Optimistic (At Least, Publicly)

In the face of these potential warning signs, Silicon Valley leadership is putting off a decidedly unconcerned face. OpenAI CEO Sam Altman remains adamant that bigger models will continue to be better, as stated in his manifesto “The Intelligence Age.” Guests and speakers at the Web Summit last week opined that AI would continue to grow as it had previously, while former Google CEO Eric Schmidt asserted that there was “no sign” that data scaling principles had yet begun to fail. But, in private, OpenAI has formed an internal task force to try and find a way forward, in the face of dwindling training data. 

All of this is compounded by a growing problem – AI is failing to prove as revolutionary and profit-generating as many had hoped. Banks are increasingly finding that AI is failing to drive profit margins, while IT firms find that it is a “tax with no clear gains.” Despite this, and reports that firms are struggling to find value in enterprise AI, corporate adoption of AI continues at a breakneck pace, with managers and executives claiming a high level of confidence in AI’s value. All the while, returns-on-investment for AI developers such as Microsoft remain distant.

Lack of Unused Training Data

One of the biggest challenges facing AI developers is the lack of unused, human-generated training data. Evidence is mounting that, if allowed to consume AI-generated content, AI models’ output will begin to degrade until the model eventually collapses outright, producing unintelligible content as the model loses any correspondence to reality. The rapid expansion of AI up to this point has been, in large part, driven by the application of the same types of LLM technology to progressively larger human-generated datasets. However, to put it simply, AI developers are running out of data that has not already been used. Human creative output is not enough to actually sustain the needs of the AI industry, posing a significant problem going forward.

We’ve Been Here Before: Concerns of a “Third AI Winter”

AI has a surprisingly long history, and its historical development is commonly divided into a series of cycles. These cycles are characterized by initial periods of expansion, as hype for the future promise of AI drives investment, which drives further advancement, generating more hype in a self-sustaining reaction until, eventually, the limits of the current generation of technology are reached, and the hype fades as businesses realize firsthand the limitations of the technology. The chain reaction then reverses, with pessimism driving cuts in funding and investment, which in turn drive reduced advancement, heightening pessimism about the technology and driving many AI development firms out of business. This creates a period of relative technological stagnation known as an “AI Winter,” before the cycle eventually repeats itself.

The first AI boom occurred in the 1960s and early 1970s, when the first generation of electronic computers led to hopes that true artificial intelligence was right around the corner. Most of the theory around artificial intelligence now existed, much of it posited decades earlier by mathematicians like Alan Turing. Now, it seemed, the only obstacle was using the power of the silicon transistor and microelectronics to make these dreams of AI a reality. But, it was not to be. Machine translation, hailed as the wave of the future, turned out to be well beyond the reach of the era’s computer technology. By the early 1970s, the failure of the technology to mature had prompted a series of crippling cuts in funding in the US and UK. The “First AI Winter” had arrived. 

It took until the early 1980s for confidence in the technology to recover to the point of another boom, as hype grew that a new generation of computer systems would finally be able to realize the potential of AI. The late 1970s had seen the emergence of the personal computer, and with these machines were now found in businesses all across the world, it seemed like the hardware challenges of the 1960s had finally been solved. Investors were convinced that new “expert systems,” coded in the Lisp programming language and designed to mimic the judgment of learned experts, would quickly find their way into the workplace and replace human expertise in a variety of fields. Leveraging the booming computer industry, startups like Symbolics and Lucid rapidly grew into multi-billion dollar businesses as they spun up production of Lisp-running computer stations. In 1985, Symbolics would register the world’s first “.com” website. 

Amid all of renewed optimism, two researchers, Roger Schank and Marvin Minsky, gave a talk at the 1984 meeting of the American Association of Artificial Intelligence, coining the term “AI Winter” to describe the funding cuts of the mid-1970s. They gave a stark warning – that unrestrained hype would lead to this cycle repeating in the future. Minsky himself was one of the co-creators of the world’s first “expert system.” But, at the time, few listened – the Lisp machines, after all, were already making their way onto the market, and the future seemed impossibly bright.

There was just one problem – the Lisp machines weren’t actually useful. In practice, they were overpriced, difficult to use, and would produce unpredictable errors when confronted with novel situations. Their programming was brittle, and incapable of adapting to new circumstances. Making matters worse, companies like Sun Microsystems had just put cheap, human-operated IBM and Microsoft computer workstations on the market. These workstations could do the same things as the Lisp machines, but for a fraction of the price. Within three years, Schank and Minsky ended up being right, and the nascent, multibillion dollar AI industry imploded. This marked the beginning of the “Second AI Winter,” lasting until the early 2010s. During this period, computer scientists tended to cloak their own products in euphemisms, trying to avoid the term “AI” and its connotations of failed promises.

The industry is, without a doubt, in an AI “spring.” The technology is booming, with advances (both positive and negative) coming at a bewildering rate. The question, therefore, is how long this can continue. The ultimate issue is whether the AI industry can innovate itself out of this cycle, by continuing to make sufficient advances to support current levels of hype and investment. (Additionally, it must find ways to make AI actually provide value for consumers – as it stands, enterprise AI faces a serious problem with actually turning a profit.) In many respects, the current generation of AI has succeeded where the previous two generations have failed, in that it has delivered an actually usable product. 

The problem, however, is that the pace of advancement up to this point is arguably unsustainable. The fact that dataset upscaling has begun to falter as a formula for creating better AI is an ominous sign for the industry. If one applies the lens of the “AI winter” cycle, the message for the industry is clear – it must innovate neural networks on a fundamental level to make better, rather than simply bigger, AI, or face a “Third AI Winter.” 

 

Matthew Sparks was a Justice Fellow at the Tech Institute 2023-2024.