Multiple reports have emerged in the last week stating that conventional \u201cupscaling\u201d – that is, the use of increasingly large datasets in an effort to produce better AI – is losing its effectiveness, and that concern has begun to mount within the AI industry. OpenAI insiders report that GPT-4\u2019s replacement, codenamed \u201cOrion,\u201d is not reliably performing better than GPT-4 itself, as prominent AI researchers warn that up-scaling is reaching its limit.<\/p>\n
This week, I\u2019m going to be examining this particular story in greater depth, as well as the history of the AI field.<\/p>\n
Bottom Line:<\/strong> Evidence is mounting that the current era of explosive, exponential AI growth may be coming to an end, as up-scaling of datasets loses effectiveness as a means to produce better AI. Though this has yet to be reflected on Wall Street, this news may bring a potential \u201cAI bubble\u201d closer to deflation.<\/span><\/i><\/p>\n <\/p>\n Several weeks ago, Reuters reported that top AI scientists, as well as sources within OpenAI, have stated that the previously-rapid progression of AI development is beginning to hit serious structural roadblocks.<\/a> Subsequently, other industry sources have made similar reports. Tech outlet <\/span>The Information<\/span><\/i> reported that, per OpenAI employees, the company\u2019s next-gen successor to the now two-year-old GPT-4, codenamed Orion, <\/span>is not reliabl<\/span>y<\/span> performin<\/span>g<\/span> better than its predecessor at certain tasks<\/span>,<\/span> and that the rate of model improvement has decreased from previous iterations.<\/span><\/a>\u00a0Bloomberg<\/span><\/i> has additionally reported <\/span>similar slowdowns are occurrin<\/span>g<\/span> with the next iteration of Goo<\/span>g<\/span>le\u2019s \u201cGemini\u201d and Anthropic\u2019s \u201cCl<\/span>y<\/span>de\u201d AI models<\/span>,<\/span><\/a> suggesting that<\/span> whatever is happening is not unique to any one LLM. <\/b>Concerns have be<\/span>g<\/span>un to spread within the industry,<\/span> <\/a>and the tech world more broadly, that a significant slowdown, or even bubble-burst, may be in the cards.<\/span><\/p>\n In the face of these potential warning signs, Silicon Valley leadership is putting off a decidedly unconcerned face. <\/span>OpenAI CEO Sam Altman remains adamant that bi<\/span>gg<\/span>er models will continue to be better<\/span>,<\/span> as stated in his manifesto \u201cThe Intelli<\/span>g<\/span>ence A<\/span>g<\/span>e.\u201d<\/span><\/a> Guests and speakers at the Web Summit last week opined that AI would <\/span>continue to <\/span>g<\/span>row as it had previously<\/span><\/a>, while former Google CEO Eric Schmidt asserted that there was <\/span>\u201cno si<\/span>g<\/span>n\u201d that data scalin<\/span>g<\/span> principles had <\/span>y<\/span>et be<\/span>g<\/span>un to fail.<\/span><\/a> But, in private, OpenAI has<\/span> formed an internal task force to tr<\/span>y<\/span> and find a wa<\/span>y<\/span> forward<\/span>,<\/span> in the face of dwindlin<\/span>g<\/span> trainin<\/span>g<\/span> data.<\/span>\u00a0<\/span><\/a><\/p>\n All of this is compounded by a growing problem – AI is failing to prove as revolutionary and profit-generating as many had hoped. <\/span>Banks are increasin<\/span>g<\/span>ly findin<\/span>g<\/span> that AI is failin<\/span>g<\/span> to drive profit mar<\/span>g<\/span>ins,<\/span><\/a> while IT firms find that it is <\/span>a \u201ctax with no clear <\/span>g<\/span>ains.\u201d<\/span><\/a> Despite this, and reports that firms are struggling to find value in enterprise AI, <\/span>corporate adoption of AI continues at a breakneck pace<\/span><\/a>, with managers and executives claiming a high level of confidence in AI\u2019s value. All the while, returns-on-investment for AI developers such as Microsoft <\/span>remain distant.<\/span><\/a><\/p>\n One of the biggest challenges facing AI developers is the lack of unused, human-generated training data. Evidence is mounting that, <\/span>if allowed to consume AI-<\/span>g<\/span>enerated content<\/span>,<\/span> AI models\u2019 output will be<\/span>g<\/span>in to de<\/span>g<\/span>rade until the model eventuall<\/span>y<\/span> collapses outri<\/span>g<\/span>ht, producin<\/span>g<\/span> unintelli<\/span>g<\/span>ible content as the model loses an<\/span>y<\/span> correspondence to reality.<\/span><\/a> The rapid expansion of AI up to this point has been, in large part, driven by the application of the same types of LLM technology to progressively larger human-generated datasets. However, to put it simply, AI developers are running out of data that has not already been used. Human creative output is not enough to actually sustain the needs of the AI industry, posing a significant problem going forward.<\/span><\/p>\n AI has a surprisingly long history, and its <\/span>historical development is commonly divided into a series of cycles.<\/b> These cycles are characterized by initial periods of expansion, as hype for the future promise of AI drives investment, which drives further advancement, generating more hype in a self-sustaining reaction until, eventually, the limits of the current generation of technology are reached, and the hype fades as businesses realize firsthand the limitations of the technology. The chain reaction then reverses, with pessimism driving cuts in funding and investment, which in turn drive reduced advancement, heightening pessimism about the technology and driving many AI development firms out of business. <\/span>This creates a period of relative technological stagnation known as an \u201cAI Winter,\u201d before the cycle eventually repeats itself.<\/b><\/p>\n The first AI boom occurred in the 1960s and early 1970s, <\/b>when the first generation of electronic computers led to hopes that true artificial intelligence was right around the corner. Most of the theory around artificial intelligence now existed, much of it posited decades earlier by mathematicians like Alan Turing. Now, it seemed, the only obstacle was using the power of the silicon transistor and microelectronics to make these dreams of AI a reality. But, it was not to be. Machine translation, hailed as the wave of the future, turned out to be well beyond the reach of the era\u2019s computer technology. By the early 1970s, the failure of the technology to mature had prompted a series of crippling cuts in funding in the US and UK. The \u201cFirst AI Winter\u201d had arrived.\u00a0<\/span><\/p>\n It took until the early 1980s for confidence in the technology to recover to the point of another boom, as hype grew that a new generation of computer systems would finally be able to realize the potential of AI.<\/b> The late 1970s had seen the emergence of the personal computer, and with these machines were now found in businesses all across the world, it seemed like the hardware challenges of the 1960s had finally been solved. <\/span>Investors were convinced that new \u201cexpert s<\/span>y<\/span>stems<\/span>,<\/span>\u201d coded in the Lisp pro<\/span>g<\/span>rammin<\/span>g<\/span> lan<\/span>g<\/span>ua<\/span>g<\/span>e and desi<\/span>g<\/span>ned to mimic the <\/span>j<\/span>ud<\/span>g<\/span>ment of learned experts<\/span>,<\/span> would quickl<\/span>y<\/span> find their way into the workplace and replace human expertise in a variet<\/span>y<\/span> of fields.<\/span> <\/a>Leveraging the booming computer industry, startups like Symbolics and Lucid rapidly grew into multi-billion dollar businesses as they spun up production of Lisp-running computer stations. In 1985, Symbolics would register the world\u2019s first \u201c.com\u201d website.\u00a0<\/span><\/p>\n Amid all of renewed optimism, two researchers, Roger Schank and Marvin Minsky, gave a talk at the 1984 meeting of the American Association of Artificial Intelligence, <\/span>coining the term \u201cAI Winter\u201d<\/b> to describe the funding cuts of the mid-1970s. They gave a stark warning – that unrestrained hype would lead to this cycle repeating in the future. Minsky himself was one of the co-creators of the world\u2019s first \u201cexpert system.\u201d But, at the time, few listened – the Lisp machines, after all, were already making their way onto the market, and the future seemed impossibly bright.<\/span><\/p>\n There was just one problem – the Lisp machines weren\u2019t actually useful.<\/b> In practice, they were overpriced, difficult to use, and would produce unpredictable errors when confronted with novel situations. Their programming was brittle, and incapable of adapting to new circumstances. Making matters worse, companies like Sun Microsystems had just put cheap, human-operated IBM and Microsoft computer workstations on the market. These workstations could do the same things as the Lisp machines, but for a fraction of the price. Within three years, Schank and Minsky ended up being right, and the nascent, multibillion dollar AI industry imploded. <\/span>This marked the beginning of the \u201cSecond AI Winter,\u201d lasting until the early 2010s.<\/b> During this period, computer scientists tended to cloak their own products in euphemisms, <\/span>tr<\/span>y<\/span>in<\/span>g<\/span> to avoid the term \u201cAI\u201d and its connotations of failed promises.<\/span><\/a><\/p>\n The industry is, without a doubt, in an AI \u201cspring.\u201d <\/b>The technology is booming, with advances (both positive and negative) coming at a bewildering rate. The question, therefore, is how long this can continue. The ultimate issue is whether the AI industry can innovate itself out of this cycle, by continuing to make sufficient advances to support current levels of hype and investment. (Additionally, it must find ways to make AI actually provide value for consumers – <\/span>as it stands<\/span>,<\/span> enterprise AI faces a serious problem with actually turnin<\/span>g<\/span> a profit.<\/span>)<\/span><\/a> In many respects, the current generation of AI has succeeded where the previous two generations have failed, in that it has delivered an actually usable product.\u00a0<\/span><\/p>\n The problem, however, is that the pace of advancement up to this point is arguably unsustainable. <\/b>The fact that dataset upscaling has begun to falter as a formula for creating better AI is an ominous sign for the industry. If one applies the lens of the \u201cAI winter\u201d cycle, the message for the industry is clear – it must innovate neural networks on a fundamental level to make <\/span>better<\/span><\/i>, rather than simply <\/span>bigger<\/span><\/i>, AI, or face a \u201cThird AI Winter.\u201d\u00a0<\/span><\/p>\n <\/p>\n Matthew Sparks was a Justice Fellow at the Tech Institute 2023-2024.<\/p>\n","protected":false},"excerpt":{"rendered":" Multiple reports have emerged in the last week stating that conventional \u201cupscaling\u201d – that is, the use of increasingly large datasets in an effort to produce better AI – is […]<\/p>\n","protected":false},"author":19629,"featured_media":0,"parent":7881,"menu_order":5,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_acf_changed":false,"_price":"","_stock":"","_tribe_ticket_header":"","_tribe_default_ticket_provider":"","_tribe_ticket_capacity":"0","_ticket_start_date":"","_ticket_end_date":"","_tribe_ticket_show_description":"","_tribe_ticket_show_not_going":false,"_tribe_ticket_use_global_stock":"","_tribe_ticket_global_stock_level":"","_global_stock_mode":"","_global_stock_cap":"","_tribe_rsvp_for_event":"","_tribe_ticket_going_count":"","_tribe_ticket_not_going_count":"","_tribe_tickets_list":"[]","_tribe_ticket_has_attendee_info_fields":false,"footnotes":"","_tec_slr_enabled":"","_tec_slr_layout":""},"class_list":["post-8819","page","type-page","status-publish","hentry"],"acf":[],"ticketed":false,"_links":{"self":[{"href":"https:\/\/www.law.georgetown.edu\/tech-institute\/wp-json\/wp\/v2\/pages\/8819","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.law.georgetown.edu\/tech-institute\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.law.georgetown.edu\/tech-institute\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.law.georgetown.edu\/tech-institute\/wp-json\/wp\/v2\/users\/19629"}],"replies":[{"embeddable":true,"href":"https:\/\/www.law.georgetown.edu\/tech-institute\/wp-json\/wp\/v2\/comments?post=8819"}],"version-history":[{"count":8,"href":"https:\/\/www.law.georgetown.edu\/tech-institute\/wp-json\/wp\/v2\/pages\/8819\/revisions"}],"predecessor-version":[{"id":8864,"href":"https:\/\/www.law.georgetown.edu\/tech-institute\/wp-json\/wp\/v2\/pages\/8819\/revisions\/8864"}],"up":[{"embeddable":true,"href":"https:\/\/www.law.georgetown.edu\/tech-institute\/wp-json\/wp\/v2\/pages\/7881"}],"wp:attachment":[{"href":"https:\/\/www.law.georgetown.edu\/tech-institute\/wp-json\/wp\/v2\/media?parent=8819"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Analysis: Growing Signs of AI Development Slowdown<\/h2>\n
Tech Execs Remain Optimistic <\/b>(<\/b>At Least<\/b>,<\/b> Publicl<\/b>y<\/b>)<\/b><\/h3>\n
Lack of Unused Trainin<\/b>g<\/b> Data<\/b><\/h3>\n
We\u2019ve Been Here Before: Concerns of a \u201cThird AI Winter\u201d<\/b><\/h3>\n