For years, Product Managers have obsessed over finding the right metrics (me included).

But we never stopped to challenge a fundamental assumption: that we were picking from a predefined set of metrics. Similarly to picking a dish from a menu, we’d choose the one that seemed right, but we wouldn’t ask the chef to create a new dish just for us. It’s simply not viable for the restaurant, just like, until recently, it wasn't really viable for PMs either. We worked with the metrics our tools could collect.

But AI has broadened our choices. We can now invent metrics that better reflect the business goals we’re trying to measure. Metrics that would have been too expensive or simply impossible to build a few years ago.

Let me show you how with an example I frequently use in my workshops.

Goals-Signals-Metrics (GSM)

One of my go-to frameworks is Google's Goals–Signals–Metrics (GSM).

I explain it in much more detail in this article but, as a quick refresher, this is how you apply it: first, define the goal you're trying to achieve. Then ask yourself what signals you would expect to observe if you were moving in the right direction. Finally, you define the metrics that quantify those signals.

Normally, this framework holds up well, but I came across a use case that forced me to bend it. Which eventually became part of my Success Metrics Workshop.

Case Study: the Indie Movie Community

A while ago, I stumbled on an indie movie community. An old-school forum, the kind that takes you back to the 1990s. Members log in, watch films shared by other members, and each film gets its own thread where people discuss it.

What was even more interesting about it was their mission statement: they wanted people to walk away from a film thinking about its main message.

Lovely mission, but a nightmare to measure. How can you possibly do that? It’s not like you can open someone's head and check whether they're thinking about a film's deeper meaning.

So the metrics lady (aka, me) had to take this on as a challenge.

I started with my normal approach for fluffy, difficult to measure goals: apply the GSM framework. The first step was easy: the Goal is to make people think about the main message of the films they watch.

Then, I identified three signals that would tell me people were engaging with that goal:

1. People discuss the movies in the community forum

2. A watcher watches more movies frequently (meaning the time between each film watched shrinks)

3. Engagement with other people’s comments in the forum in the specific channels for discussing movies (as opposed to simply posting and disappearing)

Then I translated those into metrics. By the end of the quarter, we’d check:

1. Number of movie watchers who write in the community forum per movie

2. Watchers who watch at least two films within a month of their last one

3. Percentage of all messages in the movie discussion channel that get reactions from other community members

The first two metrics looked solid in the first place. The third was meant to be a quality check on the first one, but once I thought about it properly, I realised it breaks. Engagement doesn’t necessarily relate to people thinking about the movie’s deep meaning. It might, in some cases. But it tells you nothing about the quality of the comments. Which in return made me realise that the first metric isn’t solid either. With all respect to myself, I was an idiot at this point. I teach that a pure number doesn’t reflect the relevance or quality of what we’re counting, and then I fell into the same trap. But let’s focus on metric 3 to continue the thought exercise.

A good way to think about this is to bring up my favourite platform to hate: LinkedIn. What do you see there all the time? Someone posts something purposefully for others to disagree with (or just something plain rude), and suddenly they've got a hundred reactions. None of that means the comments were insightful, it was just a shit show.

Now, when we apply the same logic to the film community, we quickly see how this metric falls apart: the percentage of messages getting reactions doesn't tell you whether people are engaging with the meaning of the film. It tells you whether people are engaging, full stop, and that’s not the same thing.

By the metric, this is a fantastic outcome. By the community's mission, it's probably a failure.

Notice that GSM didn't fail here. The framework actually did exactly what it was supposed to do: it exposed that our metric wasn't really measuring the signal we cared about.

I invented a new metric with the help of AI

Once I realised the obvious metric was misleading, I stopped looking for a better version of it and started asking a different question: what if the metric I needed simply didn't exist yet?

So I started thinking about a Discussion Depth Index.

A few years ago, this would have been impossible, or at least very difficult, not worth the ROI, as it would require a lot of manual reading and guesswork. But now, with AI, it’s easier than ever to actually read the substance of what people write, not just measure whether they reacted to it. No excuses anymore!

So, at first, I imagined an LLM could read every comment in a thread and tell me whether people were genuinely discussing the film, not just posting and reacting. That already solves the “LinkedIn problem” above: depth, not reaction count.

But then another question occurred to me: is depth enough?

Maybe I also want to know whether people are discussing the ideas the filmmaker actually intended to communicate. That seems like a natural next layer, so I thought about tagging each film with keywords representing its intended message (or better, letting the person who uploaded it state that message themselves), then checking how closely the discussion matched.

And then I caught myself again. Great films often lead different people to completely different interpretations, and that's not a failure of the film, that's often the whole point of a good one. So should disagreement with the "intended message" actually reduce the score? I don't think it should. Depth and match aren't the same thing, and treating them as one number would punish exactly the kind of discussion this community exists for.

So that's two separate dimensions, not one: how deeply people engage, and whether that engagement lines up with what the film was trying to say. And finally, there's a third worth asking, too: did people leave with any interpretation at all? Because "Great movie!" isn't one.

Ever since I’ve found this community online, this is one out of three cases in my Success Metrics That Matter in-house and public workshop that we discuss to understand how to measure anything. I love how creative people become with this case - I’ve heard lots of different versions by now of how this index could be defined, what’s important to consider for this index to work, how to actually implement it, and how else the community’s mission could be measured. It frees up the brain from what WAS possible and opens it up for what IS possible now.

The takeaway

None of these metrics exist in Google Analytics, or come pre-built in Mixpanel or Amplitude. For a long time, we've designed products around the metrics our tools made easy to collect. AI gives us the opportunity to reverse that process. We can start with the outcome we genuinely care about and then ask ourselves, "If we had unlimited ability to understand text, conversations or images, how would we choose to measure success?"

That feels to me much more exciting than using AI to summarise customer interviews (not that this ability is not exciting). It's the possibility of creating measurements that are finally aligned with the thing we're actually trying to achieve.

The indie movie community is one of three examples I use in my Success Metrics workshop to show how Goals-Signals-Metrics can help teams tackle measurement problems that initially seem impossible. If that's the sort of thing your team needs to get better at, or simply making your team understand how to measure success of what they build in a meaningful way, reach out and I'll tell you more about the workshop.

For now, I’ll leave you with a challenge: think about the last goal your team called "unmeasurable" and gave up on. What was it, what metric did you never build for it, and what metric would you build now with the help of AI? Now is the right time to invent that metric.

—

This article was edited by Diana Bernardo.