A reader recently sent me two questions related to her day-to-day work challenges. I answered the first one a few weeks ago, and today we’ll dive into the second one:

"Another challenge I see is to transfer qualitative feedback into metrics, e.g. we get a lot of different user feedback from customer service, interviews, etc., and they all sound kind of different. What do I do with it?"

This reader works at an agency, but this is far from being a service provider problem. I’ve worked with many internal product teams that struggled with the same at some point.

My answer to this question has two parts.

Part 1: Let a Tool Do the Heavy Lifting

Most teams collect feedback from multiple sources. That’s the right approach!

But then, how do you combine customer service tickets, interviews, and surveys, for example? The first step is to make sense of it in an automatic, scalable way. The best way I know is to use tools specifically built for this, like Usersnap (European) or Dovetail that allow you to centralise feedback, tag themes, and identify patterns without having to read every response three times.

An alternative that also works: stitch something together yourself.

For example, when I worked at Doodle, we built an automation with Zapier that pulled qualitative feedback from different sources into Airtable, generated heat maps, wrote summaries, and helped us prioritise. Was it perfect? I wish… But, at least, it gave us something to work with.

👉 Psst - if you're looking for European alternatives: use Softr Workflows, n8n or make instead of Zapier and SeaTable or Baserow instead of Airtable. Not affiliated. Just a hint. 👈

Or give AI access to your knowledge base like Obsidian and treat that knowledge base be place to collect all feedback from different sources. Follow Else van der Berg to learn more about using AI for analysing qualitative feedback.

The point is: don't let the volume of feedback become the reason you ignore it.

Part 2: Quantify What You’re Seeing

Once you have the feedback in one place, you can start the real analysis. Sounds easier, right? Well, this is actually where I see most teams getting stuck because qualitative feedback feels impossible to measure.

Good news: it isn't.

When you're running customer interviews (say, 12 of them), you're never going to have statistical significance. That's not a problem - what you're looking for is pattern and proportion. If 6 out of 12 people mention the same friction, you're onto something. That's how you quantify qualitative feedback: X out of Y say or do the same thing. Or are willing to do Z. Or cannot do whatever. Etc.

The same logic applies to usability testing. If you change a flow and want to know whether it's better, track how many people get stuck at a specific step. If today 4 people get stuck and after your change it's 2, you know it worked. If it's still 4, or it rose to 6, keep what you had before (you don’t want to change a working system).

One thing to keep in mind across all of this: don't only measure whether the change worked. Also measure whether everything else that was important stayed intact, aka set up some health metrics.

Finally, a note on hypothesis-based experimentation. If you're testing a solution, define upfront what success looks like, in concrete terms. As an example: “if we try this X times and achieve Y result in Z% of cases, we know it works.” This way, you don’t need statistical significance, you only need to know if you’ve achieved the percentage you defined upfront.

Still Not Sure What To Do With Your Specific Case?

You can approach this in different ways, depending on your context.

If you’re still struggling after implementing this, reach out to me and tell me more about your situation. I'll do my best to help (it might even turn into another newsletter, if it’s a situation many people struggle with!)

And if you want to go even further on the topic of turning fuzzy goals into measurable outcomes, I wrote about Goal-Signal-Metrics in an earlier newsletter. Read it here.