The Value of Asking Users Qualitative Questions As A Product Manager
Asking the right questions is one of the most important aspects of being a Product Manager.
I would put it right up there with empathy as arguably the most important skill to have as a Product Manager.
Ask the wrong kinds of questions, and you’ll gain misguided insights and make decisions that don’t align with what your users actually want. To quote a math teacher I had in High School:
You’re on the right track…but the wrong train!
There are many kinds of questions that you can ask that can potentially lead you in taking the wrong train.
When you use questions that are leading, biased, or loaded, questions with yes or no answers, or 1-5 scale answers, you may get answers that you want, but not the ones you need. You are also missing out on a goldmine of insight.
To clarify, there is still plenty of value to gain from quantitative questions.
You may have hundreds, thousands, or millions of responses to these questions, and it’s very easy to quantify them to gain insights. However, you can’t just rely on quantitative questions to gain insight into how users are using your product.
Qualitative questions requires you to sit down and go through answers. Taking time to sit down and read these answers can give you incredible insights on how people view and use your product.
I want to discuss a qualitative question that I use to gain high levels of insights from users.
Here is the qualitative question I like to ask:
What are some of your biggest challenges or pain points with using X?
That’s it. Nothing fancy.
What makes the “biggest challenges” question so effective?
It’s an open-ended question where you’ll receive open ended answers.
You get to the heart of the matter, and you find out what are some of the biggest pain points that your customers are having.
You get what you ask for.
It’s easy to ask users questions that will give you answers that you were hoping to get (a biased/leading question). With the biggest challenges/pain points question, you get what you ask for, and may get answers you didn’t want to hear, but needed to. Perhaps you’ll learn that a lot of people think your product sucks and are forced to use it at their work because it was mandated by the higher-ups.
Customers love to talk about their challenges, their concerns, their pain points, and if you give them the opportunity to, they will.
To clarify once again, there is nothing wrong with quantitative questions. You should be asking them!
They are very valuable. You can check over time how customers opinions are changing. Perhaps surveys have indicated more people are likely to recommend your product compared to a year ago. However, it’s easy to get into these points of data that can simply be ego stroking, and ignore the heart of the matter and finding the pain point your customers have. You need to supplement them with qualitative questions as well.
Asking the “biggest challenges” question makes you very vulnerable. You’ve asked your customers to identify the biggest struggles that they face with your product.
So, what do most people and companies do? They avoid asking these kind of qualitative questions. They fall back on quantitative questions and metrics that are easy to measure.
I’ve worked with companies or clients in the past that were reluctant to ask this kind of question, but they were glad when they did.
One client was far too focused on metrics NPS and avoided talking to their customer, and once they reached out to them with an open ended question, they began to see just how off they were with what their customers wanted and what they thought the customers wanted, and were able to course correct with these insights they gained from asking open ended questions.
Asking more qualitative questions is one of the best things you can do as a product manager to gain insights into what your users think of your product. Open ended answers take time to go through, but it’s worth your time to sit down and read through them. In the past, I’ve seen users responses range from “It’s alright” to a user writing War and Peace about their challenges with a product. Don’t be shy about throwing more qualitative questions into the mix.