Google's Gemini Ultra AI model may have cost $191 million, new Stanford AI report estimates (2024)

Overall, the report paints a picture of a rapidly growing and increasingly complex (and expensive) AI landscape dominated by commercial entities, particularly U.S. tech giants. The number of new LLMs released globally in 2023 doubled compared to the previous year, according to the report. Investment in generative AI also skyrocketed, and so did global mentions of AI in legislative proceedings and regulations—in the U.S. alone last year, the total number of AI-related regulations increased by 56.3%.

One of the biggest takeaways from the report, however, is the dominance of U.S. tech companies. While two-thirds of models released last year were open-source, the highest-performing models came from commercial entities with closed systems. Private industry accounted for 72% of foundational models released last year, putting out 108 compared to 28 from academia and just four from government. Google alone released a whopping 18 foundation models in 2023. For comparison, OpenAI released seven, Meta released 11, and Hugging Face released four. Overall, U.S. companies released 62 notable machine learning models last year compared to 15 coming out of China, eight from France, five from Germany, and four from Canada.

(U.S. dominance also comes across clearly in data on funding rounds for generative AI companies that Fortune published earlier this week.)

The main reason for this dominance is made crystal clear by new cost estimates for model training included in the report.

“The training costs of state-of-the-art AI models have reached unprecedented levels,” it reads, citing the exponential increase as a reason academia and governments have been edged out of AI development.

According to the report, Google’s Gemini Ultra cost an estimated $191 million worth of compute to train, and OpenAI’s GPT-4 cost an estimated $78 million, which is actually slightly lower than some previous estimates of how much that model cost. (Now imagine how much more it’d be if these companies had to pay for all the training data they scraped from the internet.) For comparison, the report notes that the original 2017 Transformer model, which introduced the architecture underlying all of today’s LLMs, cost only around $900.

On the achievements and potential of AI, the report discusses how AI systems have passed human performance on several benchmarks—including some in image classification, visual reasoning, and English understanding—and what it’s doing to turbocharge scientific discovery. While AI started to accelerate scientific discovery in 2022, 2023 saw the launch of even more significant science-related AI applications, the report says. Examples include Google DeepMind’s GNoME (an AI tool that facilitates the process of materials discovery—although some chemists have accused the company of overstating the model’s impact on the field), EVEscape (an AI tool developed by Harvard researchers that can predict viral variants and enhance pandemic prediction), and AlphaMissence (which assists in AI-driven mutation classification).

AI systems have also demonstrated rapid improvement on the MedQA benchmark test for assessing AI’s clinical knowledge. GPT-4 Medprompt, which the report calls “the standout model of 2023” in the clinical area, reached an accuracy rate of 90.2%—marking a 22.6% increase from the highest score in 2022. What’s more, the FDA is approving more and more AI-related medical devices, and AI is increasingly being used for real-world medical purposes.

Of course, AI progress is not a straight line, and there are many significant challenges, lingering questions, and legitimate concerns.

“Robust and standardized evaluations for LLM responsibility are seriously lacking,” the report authors wrote, citing how leading AI developers primarily test their models against different responsible AI benchmarks, complicating efforts to systematically compare the risks and limitations of the top models.

The report highlights many other issues surrounding the technology: Political deepfakes are simple to create but difficult to detect; the most extreme AI risks are difficult to analyze; there is a lack of transparency around the data used to train LLMs and around key aspects of their specific designs; researchers are finding more complex vulnerabilities in LLMs; ChatGPT is politically biased (toward Democrats in the U.S. and the Labour Party in the U.K.); and LLMs can output copyrighted material. Additionally, AI is leaving businesses vulnerable to new privacy, security, reliability, and legal risks, and the number of incidents involving the misuse of AI is rising—2023 saw a 32.3% increase over 2022.

Clocking in at over 500 pages, the report is a doozy. But it’s unquestionably the deepest and most thorough overview of the current state of AI available at the moment. If you want to dive deeper but don’t have time for the full report, HAI has also published some handy charts and will be presenting the findings and answering questions in a webinar on May 1.

And with that, here’s more AI news.

Sage Lazzaro
sage.lazzaro@consultant.fortune.com
sagelazzaro.com

AI IN THE NEWS

Meta rolls out its Llama 3 open-source AI models. That's late-breaking news, as the tech giant released the first new models from the Llama 3 family. The company released two smaller versions of the models that the company says are the highest-performing open-source models in those size ranges in the market. It also said it is still training a much larger, 400 billion parameter version that is designed to match the capabilities of some of the best models from OpenAI, Anthropic, and Google, but that it will decide when and how to release it following safety testing in the coming months. My Fortunecolleague Sharon Goldman has the skinny on Llama 3 and why the new models are debuting in a very different environment for open-source AI than it was when the previous version of Llama was released last summer. You can read her analysis here.

The EU decides not to formally investigate Microsoft over its OpenAI deal. That’s according to Bloomberg. EU regulators determined the tech giant’s investment into OpenAI doesn’t merit a formal probe because it falls short of a takeover and there’s no evidence of Microsoft controlling the company. The EU’s antitrust arm said in January that it was reviewing if a formal investigation was needed, sparking concerns that Microsoft and OpenAI might be forced to unwind their partnership.

Meta’s oversight board is set to investigate two deepfake p*rn cases. That’s according to Wired. The cases involve unnamed celebrities whose images were altered to create explicit content that was circulated on the company’s platforms. The use of AI to create nonconsensual explicit content is becoming extremely common, and as stated in the Stanford HAI report, these materials are easy to generate but difficult to detect. Women and girls are overwhelmingly targeted, including both public figures like Taylor Swift and Rep. Alexandria Ocasio-Cortez (D-N.Y.) and everyday people. Last month, a Beverly Hills middle school expelled five students who used generative AI to create fake nude images of classmates.

OpenAI releases updated assistants API aimed at business customers. The AI company released an improved version of the tool, which lets enterprise customers build digital assistants that can help automate certain business processes based on its GPT models. The new version allows each agent to handle up to 10,000 separate files, with automatic chunking and embedding of their data, allowing the assistant to conduct searches, and summarize the documents. There are also more controls to help enterprises manage how much the assistant costs and which tools it can access and use. You can read more on this OpenAI thread on X.

Stability AI lays off employees, according to Business Insider report. The publication said that the once-hot AI startup, whose founder Emad Mostaque stepped down from its board last month following a dispute with investors over the startup’s management, was letting go of an unspecified number of employees. It quoted an internal email it had obtained. The email was sent by Stability’s chief operating officer Shan Shan Wong and chief technology officer Christian Laforte and said the company needed to “right-size parts of the business” and “focus” its operations.

Stability AI makes Stable Diffusion 3 available to more developers. That’s according to The Verge. The model is still in preview, but now more developers will be able to access the company’s newest text-to-image model through both an API and new content creation platform. The new platform, called Stable Assistant Beta, will allow paying subscribers to access Stable Diffusion 3 and other models.

Snap plans to add watermarks to AI-generated images created with its tools. That’s according to TechCrunch. The watermark—which will appear as a translucent version of the Snap logo with a sparkle emoji—will be added to any media created with its AI lenses when they’re shared in Snapchat, exported, or downloaded to the camera roll. As I’ve previously reported, the use of labels and watermarks to designate AI content is causing more confusion than clarity.

FORTUNE ON AI

The U.S. and China are leading the world in AI innovation–but the U.K. can punch above its weight. Here’s how —Victor Riparbelli (Commentary)

How Moderna’s CIO helps steer the drugmaker’s post-COVID evolution —John Kell

Junior analysts, beware: Your coveted and cushy entry-level Wall Street jobs may soon be eliminated by AI —Sydney Lake

Fashion giant Shein has been slapped with yet another lawsuit alleging copyright infringement, data scraping, and AI to steal art: ‘It’s somewhat shocking that they’ve been able to get away with it’ —Sasha Rogelberg

While streaming giants slash budgets and users flock from X, the U.K.’s biggest publishers are using AI (with a little help from Prince Harry) to get us hooked back on books —Molly Flatt

AI CALENDAR

April 24: Meta reports calendar Q1 earnings

April 25: Microsoft and Alphabet report calendar Q1 earnings

May 7-11: International Conference on Learning Representations (ICLR) in Vienna

May 14: Google I/O conference

May 14: Stanford HAI’s RAISE Health Symposium

May 21-23: Microsoft Build conference in Seattle

June 5: FedScoop’s FedTalks 2024 in Washington, D.C.

June 25-27: 2024 IEEE Conference on Artificial Intelligence in Singapore

July 15-17: Fortune Brainstorm Tech in Park City, Utah (Register here.)

July 30-31: Fortune Brainstorm AI in Singapore. (Register here.)

Aug. 12-14: Ai4 2024 in Las Vegas

EYE ON AI RESEARCH

The robots are coming. MIT Technology Review published a great story about how various researchers are using generative AI to make serious strides in robotics. One robot has given a man permanently affected by a stroke a new level of autonomy, allowing him to control a computer mouse, brush his teeth, and play with his granddaughter. Another team of researchers is getting closer to achieving the kind of home robot that could take over all our least favorite chores. There’s also a multimodal robot that can accept prompts in the form of text, image, video, robot instructions, or measurements. Read it here.

EYE ON AI NUMBERS

3x

The number of companies with a chief AI officer (CAIO) has almost tripled globally in the past five years, according to LinkedIn data, as reported by the Financial Times. The position gained further steam in the U.S. government last month when the White House mandated federal agencies to hire CAIOS, who will convene regularly as part of a new Chief AI Officer Council. Amid the growing AI hiring frenzy, the role is commanding folks with backgrounds in both computer science and business administration. In the report, the Financial Times dives into the CAIO role at Accenture, Dell, and others.