When AI stops being a toy

If you’ve worked with AI, you’ve probably felt this gap too. Budgets are crazy high, dashboards get damn pretty, we celebrate AI usage metrics, but the impactful business metrics barely move.

Adoption is not the same as impact. The hard question for me isn’t “can it do this?” but “does it change what matters?”

According to the State of AI in Business 2025 report by MIT NANDA, 95% of businesses that invest in AI fail to generate measurable business value in return. That's crazy!

Below are the five conditions I’ve found essential when looking at AI features. Miss one and you may still ship a clever feature, but get them all right and the work can become leverage and impact.

Start with a costly, specific problem (always)

Most AI projects within established organizations begin with a capability looking for a problem.

Someone says, “We should use AI to summarise calls” or “Let’s bolt on a chatbot because everyone else has one.” Or my favorite: “Let the user only do stuff via chat + agent.”

Then a product team ships a prototype, pats themselves on the back, and nothing happens. Too hard to implement in the end.

Flip the script. Start with a user (or business) problem that hurts a lot. Where exactly are you bleeding cash or time? What is the most important unsolved problem of your user? Is there a manual compliance check that drags on for days? Do reps spend hours stitching information across systems just to respond to a ticket?

That is your starting point.

Ask yourself: What does this cost us today? How often does it happen? What’s at risk if we don’t fix it?

If you can’t answer those questions, you don’t have a problem worth solving. When you do have them, every model choice and data decision ties back to a clear outcome.

You might think this is true for every new feature or initiative, but when it comes to AI, most product managers ignore the basics—so that’s our starting point.

Embed AI into the flow of work

Too many tools sit beside the real work of your end users. They generate outputs but don’t change how decisions are made. An assistant drafts replies, but the agent still rewrites everything manually. A model scores leads, but the salesperson ignores it because it’s not part of their CRM.

Go back to the drawing board and deeply understand the job of your users and the job map to perfectly understand where AI can 10× or 100× the process—and where it might hinder what is actually working.

In the end, the dots have to connect; otherwise, real adoption is never happening. To drive impact, AI must live where the work happens.

For example:

• The model runs automatically as part of the process.
• The output directly shapes what happens next.
• There’s no way to bypass it without noticing.

Mapping this out can be uncomfortable. It forces questions and decisions about trust, accountability, and error handling. It also creates the space for feedback loops, which is how the system gets better instead of just being impressive once.

Measure outcomes, not activity

My personal favorite.

Activity metrics are seductive. DAU goes up, queries explode, automation increases. None of that matters if customer satisfaction stagnates, support costs stay flat, or revenue doesn’t budge. Think about the second-order effects of what you are introducing.

To properly track this, you need a deep understanding of how your users and the business define a positive output and outcome from your service.

Then define your inputs and strategy from there. Hold every change accountable to those metrics.

If the numbers don’t move, the AI is a toy.

Clear metrics expose hard trade-offs. You’ll cut features that feel cool but don’t move the needle, and double down on those that do.

Go where the returns are, not where the applause is

Shiny AI gets clicks and applause: flashy dashboards, generative assistants, chatbots with personality. They’re fun to show on stage, but they rarely affect the business at scale.

The highest returns often hide in boring processes—invoice reconciliation, contract review, supply chain matching. They consume headcount and introduce delays. Automating them frees people to do higher-order work and drops costs immediately.

If you’re tempted by something that is not obviously a painkiller, ask yourself: how much money does this save? How much time does it free up? If the answer is “not much,” resist the urge.

Redesign roles and accountability

Add AI to a workflow without looking at the roles that your users and the business need to play, and you’ll just create friction.

If the system makes a recommendation, who’s responsible if it’s wrong? If AI reduces someone’s workload, what happens to their role? Without a clear answer here, you might run into bigger problems and users lose trust.

You basically treat AI as a new team member.

The goal isn’t to replace people. It’s to pair them with systems in ways that amplify each other. That takes deliberate (service) design.

Pulling it all together

Most AI initiatives fail because they’re treated as features bolted onto existing structures. The ones that succeed treat AI (like agents) as part of the system:

• They solve a real, costly problem
• They embed the solution into the workflow
• They measure success in outcomes, not activity
• They invest where the returns are highest
• They redefine roles to support the new way of working

If you’re serious about creating impact, start there.

Otherwise, be honest: you’re doing AI theatre.