Report: Massive amounts of $$$ still being pumped into AI

AI continues to improve – at least according to benchmarks. But the promised benefits have largely yet to materialize while models are increasing in size and becoming more computationally demanding, and greenhouse gas emissions from AI training continue to rise.

These are some of the takeaways from the AI Index Report 2025 [PDF], a lengthy and in-depth publication from Stanford University's Institute for Human-Centered AI (HAI) that covers development, investment, adoption, governance and even global attitudes towards artificial intelligence, giving a snapshot of the current state of play.

In terms of performance, the researchers state that AI models are increasingly mastering new and challenging benchmarks designed to test their capabilities, including MMMU, GPQA, and SWE-bench. With the latter, which measures success in solving actual coding problems from GitHub, AI systems managed just 4.4 percent in 2023, but this jumped to 71.7 percent last year.

According to HAI, last year's AI Index highlighted that many models had already surpassed human performance on a range of tasks, with only a few exceptions, such as competition-level mathematics and visual commonsense reasoning. This trend largely continued over the past year, with models closing performance gaps and matching or exceeding humans on even more demanding benchmarks.

If you think that sounds depressing, the report also stresses that complex reasoning is still out of reach for AI models. Even with mechanisms such as chain-of-thought reasoning to boost their performance, large language models (LLMs) are unable to reliably solve problems for which a solution can be found using logical reasoning, making them unsuitable still for many applications.

However, HAI highlights the enormous level of investment still being pumped into the sector, with global corporate AI investment reaching $252.3 billion in 2024, up 26 percent for the year. Most of this is in the US, which hit $109.1 billion, nearly 12 times higher than China's $9.3 billion and 24 times the UK's $4.5 billion, it says.

Most companies that report financial impacts from using AI within a business function estimate the benefits as being at low levels

Despite all this investment, "most companies that report financial impacts from using AI within a business function estimate the benefits as being at low levels," the report writes.

It says that 49 percent of organizations using AI in service operations reported cost savings, followed by supply chain management (43 percent) and software engineering (41 percent), but in most cases, the cost savings are less than 10 percent.

When it comes to revenue gains, 71 percent of respondents using AI in marketing and sales reported gains, while 63 percent in supply chain management and 57 percent in service operations, but the most common level of revenue increase is less than 5 percent.

The report claims, "AI is beginning to deliver financial impact across business functions, but most companies are early in their journeys" - an excuse we've been hearing for some time now.

Meanwhile, despite the modest returns, the HAI report warns that the amount of compute used to train top-notch AI models is doubling approximately every 5 months, the size of datasets required for LLM training is doubling every eight months, and the energy consumed for training is doubling annually.

This is leading to rapidly increasing greenhouse gas emissions resulting from AI training, the report finds. It says that early AI models such as AlexNet over a decade ago caused only modest CO₂ emissions of 0.01 tons, while GPT-4 (2023) was responsible for emitting 5,184 tons, and Llama 3.1 405B (2024) pumping out 8,930 tons. This compares with about 18 tons of carbon a year the average American emits, it claims.

While North America maintains its leadership in organizations' use of AI, Greater China demonstrated one of the most significant year-over-year growth rates

To set against this, the energy efficiency of infrastructure is increasing by 40 percent annually, while hardware performance, as measured in 16-bit floating-point operations, grows by 43 percent annually, doubling every 1.9 years.

According to HAI, the US and China are vying for AI leadership, with America as the primary source of models, producing no fewer than 40 during 2024. However, China leads in AI research publication totals, while the United States leads in influential research, with the most cited AI publications.

"While North America maintains its leadership in organizations' use of AI, Greater China demonstrated one of the most significant year-over-year growth rates," the researchers pointed out.

The performance gap between models produced by the US and China is also shrinking, with American models significantly outperforming their Chinese counterparts in 2023, a trend that the HAI report says no longer holds.

Finally, a chapter of the report looks at public perception and sentiment surrounding AI, and suggests there is growing public anxiety about the technology. It says that two thirds of people now expect that AI-powered products and services will significantly impact daily life within the next 3 to 5 years.

But readers of The Register may be surprised to learn that optimism about AI is growing in areas that were previously the most skeptical about it, including the US and UK. In 2022, about 38 percent of Brits and 35 percent of Yanks viewed AI as having more benefits than drawbacks, the HAI asserts. By 2024, those numbers had risen by 8 and 4 percentage points, respectively.

• Copyright-ignoring AI scraper bots laugh at robots.txt so the IETF is trying to improve it
• AI entrepreneur sent avatar to argue in court – and the judge shut it down fast
• Canada OKs construction of first licensed teeny atomic reactor
• Meta accused of Llama 4 bait-and-switch to juice AI benchmark rank

As far as workers go, 60 percent of respondents believe that AI will change how individuals do their jobs in the next five years, while 36 percent expect that AI will replace their jobs in the next five years.

Overall, AI is viewed as a potential time saver, but not many people share the AI industry's massive hype about the economic benefits. While 55 percent think the technology will save time, only 38 percent expect AI to improve health outcomes, 36 percent expect AI will improve their national economy, and only 31 percent see a positive impact on the jobs market.

All this is little more than a summary of the AI Index report, which spans more than 450 pages and is festooned with charts and graphs to illustrate its many points. HAI says the data it is based upon is broadly sourced and is available via its website. ®