I Asked 4 Executives How They Measure AI ROI. None Started With AI Tokens.

Silicon Valley has spent the previous 12 months turning AI utilization right into a scoreboard.

However after asking 4 executives about how they measure AI return on funding at Mistral AI’s summit in Paris final month, I seen one thing putting: none of them began by speaking about what number of AI tokens workers had been utilizing.

Charles Holive, chief AI officer at BNP Paribas CIB, mentioned his staff focuses on outcomes fairly than what he known as “vanity metrics.”

“We attempt to go away from self-importance metrics — billions of tokens per day,” he mentioned. “We attempt to ensure that what we observe is an consequence, not an arrogance metric.”

As a substitute of asking what number of AI tokens workers have used, Holive mentioned he asks: “What did you do, you did not do earlier than? How a lot sooner did you do it?”

Antoine Pichot, director of innovation, digital and information at La Banque Postale, informed Enterprise Insider the financial institution measures AI by whether or not it makes workers extra environment friendly, improves customer support, and delivers worth for cash.

Amit Kapur, chief AI and transformation officer at Tata Consultancy Providers, one of many world’s largest IT providers and consulting companies, mentioned his focus is on whether or not AI is bettering enterprise efficiency fairly than simply what number of tokens workers eat.

In the meantime, Sujay Bhattacharya, NTT DATA’s govt managing director and international head of Digital Office Providers and Mistral AI, who helps giant firms undertake AI instruments from a number of AI suppliers, mentioned his prospects are more and more trying past token counts and specializing in the general value and enterprise worth of AI tasks.

The ‘tokenmaxxing’ backlash

The feedback at Mistral’s AI summit come as some US firms are starting to shift away from “tokenmaxxing” — the concept better AI use and better token consumption routinely result in greater productiveness.

Amazon shut down an inner AI-use leaderboard final month after workers started performing duties merely to climb the rankings.

Uber COO Andrew Macdonald has publicly questioned whether or not greater AI spending is producing extra useful products, saying it stays tough to attract a direct line between token consumption and buyer worth.

In the meantime, OpenAI, Anthropic, and GitHub have all moved towards usage-based pricing for companies, placing extra strain on workers at buyer firms to show that greater AI utilization is delivering significant returns.

The shift away from tokenmaxxing doesn’t suggest firms have stopped monitoring AI token utilization.

Holive mentioned his staff at BNP Paribas CIB continues to observe token consumption to regulate prices and measure adoption.

However the executives I spoke to in Paris all appeared to agree on one factor: whereas tokens could let you know how a lot AI is getting used, they do not essentially let you know whether or not it is yielding a tangible return on funding.