Snorkel AI Raises $100 Million To Build Better Evaluators For AI Models

Alex Ratner, CEO of Snorkel AI remembers a time when data labeling —the grueling task of adding context to swathes of raw data and grading an AI model’s response— was considered “janitorial” work among AI researchers. But that quickly changed when ChatGPT stunned the world in 2022 and breathed new life (and billions of dollars) into a string of startups rushing to supply human-labeled data to the likes of OpenAI and Anthropic to train capable models.

Now, the crowded field of data labelling appears to be undergoing another shift. Fewer companies are training large language models from scratch, leaving that task instead to the tech giants. Instead, they are fine-tuning models and building applications in areas like software development, healthcare and finance, creating demand for specialized data. AI chatbots no longer just write essays and haikus; they’re being tasked with high stakes jobs like helping physicians make diagnoses or screening loan applications, and they’re making more mistakes. Assessing a model’s performance has become crucial for businesses to trust and ultimately adopt AI, Ratner said. “Evaluation has become the new entry point,” he told Forbes.

That urgency for measuring AI’s abilities across very specific use cases has sparked a new direction for Snorkel AI, which is shifting gears to help enterprises create evaluation systems and datasets to test their AI models and adjust them accordingly. Data scientists and subject matter experts within an enterprise use Snorkel’s software to curate and generate thousands of prompt and response pairs as examples of what a correct answer looks like to a query. The AI model is then evaluated according to that dataset, and trained on it to improve overall quality.

The company has now raised $100 million in a Series D funding round led by New York-based VC firm Addition at a $1.3 billion valuation— a 30% increase from its $1 billion valuation in 2021. The relatively small change in valuation could be a sign that the company hasn’t grown as investors expected, but Ratner said it’s a result of a “healthy correction in the broader market.” Snorkel AI declined to disclose revenue.

Customer support experts at a large telecommunication company have used Snorkel AI to evaluate and fine tune its chatbot to answer billing related questions and schedule appointments, Ratner told Forbes. Loan officers at one of the top three U.S. banks have used Snorkel to train an AI system that mined databases to answer questions about large institutional customers, improving its accuracy from 25% to 93%, Ratner said. For nascent AI startup Rox that didn’t have the manpower or time to evaluate its AI system for salespeople, Snorkel helped improve the accuracy by between 10% to 12%, Rox cofounder Sriram Sridharan told Forbes.

It’s a new focus for the once-buzzy company, which spun out of the Stanford Artificial Intelligence Lab in 2019 with a product that helped experts classify thousands of images and text. But since the launch of ChatGPT in 2022, the startup has been largely overshadowed by bigger rivals as more companies flooded the data labelling space. Scale AI, which also offers data labeling and evaluation services, is reportedly in talks to finalize a share sale at a $25 billion valuation, up from its $13.8 billion valuation a year ago. Other competitors include Turing, which doubled its valuation to $2.2 billion from 2021, and Invisible Technologies, which booked $134 million in 2024 revenue without raising much from VCs at all.

Snorkel has faced macro challenges too: As AI models like those powering ChatGPT got better, they could label data on a massive scale for free, shrinking the size of the market further. Ratner acknowledged that Snorkel saw a brief period of slow growth right after OpenAI launched ChatGPT and said enterprises had paused pilots with some vendors to consider using AI models for labelling directly. But he said Snorkel’s business bounced back in 2023 and has grown since.

Ratner said Snorkel’s differentiator is its emphasis on bringing in subject matter experts — either its own or those within a company– and using a proprietary method called “programmatic labelling,” to automatically assign labels to massive troves of data through simple keywords or bits of code as opposed to doing it manually. The aim is to help time-crunched experts like doctors and lawyers label data faster and more economically.

As it leans into evaluation, which also requires data generation, Snorkel has started hiring tens of thousands of skilled contractors like STEM professors, lawyers, accountants and fiction writers to create specialized datasets for multiple AI developers, who then use the datasets to evaluate their models (he declined to say which frontier AI labs Snorkel works with). They can also use this data to add new functionality to their chatbots, like the ability to break down and “reason” about a difficult query or conduct in-depth research on a topic, Ratner said.

But even when it comes to building specialized evaluations, Snorkel faces fierce competition— new and old. The top AI companies have released a number of public benchmarks and open source datasets to evaluate their models. LMArena, a popular leaderboard for evaluating AI model performance, recently spun out as a new company and raised $100 million in seed funding from top investors at a hefty $600 million valuation, according to Bloomberg. Plus, companies like Scale, Turing and Invisible, all offer evaluation services. But Ratner said that unlike its rivals, Snorkel was built around human experts right from the start.

Saam Motamedi, a partner at Greylock who participated in the round, said these new specialized dataset services are a fast-growing part of Snorkel’s business as the industry shifts to what’s called “post training” — the process of tweaking the model’s performance for certain applications. AI has already soaked up most of the internet data, making datasets custom-made by domain experts even more valuable. “I think that market tailwind has proven to be a really good one for Snorkel,” he said.

MORE FROM FORBES

ForbesThis Top VC Wants To Use Main Street America As An AI LabForbesPokemon Go Made Niantic Billions. Now It’s Ditching Gaming For AI.By Richard NievaForbesAI Startup Decagon In Talks To Raise $100 Million At A $1.5 Billion ValuationBy Rashi Shrivastava