Product roadmaps are often influenced by the ‘loudest voice’, ‘the most recent voice’ (i.e. recency bias; read this post for more information), or even ‘the customer the CEO visited last’ bias. But it shouldn’t be. As the owner of the roadmap, your duty as a Product Manager is to be as objective and evidence-based as possible.
What do we mean by ‘evidence-based’? In essence, we are collecting examples of customer problems and challenges. If we can amass enough examples and analyze them collectively, we can turn isolated events into data. And while the quality of this data may be less than other types of data – for example, a statistically-significant empirical measurement – the quality of the data (and the decisions that you can make from it) improves as you collect more of it.
But how do you actually do this? It doesn’t have to be overly complex, and it doesn’t require fancy or expensive tools.
These are the main elements of the approach:
- A way to capture the evidence.
- A method of grouping the evidence.
- A routine that includes regular review and theming of the evidence.
- A process to advance the problem themes to begin solutioning and communicate these results to stakeholders. (For more about stakeholder management, see our recent post about “Decision-Driven Stakeholder Management”).
Let’s look at each element in more detail.
Capture The Evidence
The core of this approach is a tool that can capture and store your evidence long-term. This could be one of the many cloud-based tools available for this purpose – Jira, ProductBoard, Aha!, etc.) – or simply a spreadsheet with links.
The key part of the tool is that it can capture or link to the original evidence – an email, audio or video recording, or screen recording – permanently. The best evidence preserves the original inputs without summarization, paraphrasing or interpretation.
In capturing the evidence, do your best to get the right data (see this post for more information about how to objectively capture customer requests).
Group The Evidence
Once you have a system for storing and working with the evidence, the next step is to create a method of analyzing and ‘theming’ it. This is a key step because it enables you to begin identifying trends in the sea of data that you are likely to accumulate – and these trends are what you can ultimately act upon.
But how do you theme the evidence? First, let’s be clear on how not to do it: You should not theme evidence by solution. The reason is that you can’t know the right solution before you know the problem you are trying to solve. And you can’t know the problem you need to solve until you find the trends.
Some good examples of theming by problem:
- A user asks to export their data from your system in order to integrate it with data from another service.
Theme/Grouping: Integration with service XX. - A user requests a thicker bracket in order to install the product in a particular thick wall in their home.
Theme/Grouping: Installation process - thick walls.
In the first example, a possible solution – and the solution requested by the customer – is an ‘Export’ function. But is that the best solution? It is too early to tell. Instead, you must focus on identifying the problem here. Solutions come later.
Make It Routine
Now comes the important part: don’t let this work be a one-off effort.
The part of this method that makes it especially powerful, however, is doing it routinely – weekly, monthly, or even quarterly depending on the volume. From consistently theming the incoming information, you can begin to identify the trends in how people are using (or misusing) your product. Returning to our examples, you might identify the themes as follows:
- We’ve received eight requests in the past month in which the user was attempting to integrate the data from our service with that of service XX.
- In the past 90 days, we have had 12 users report difficulty installing the product in homes with walls thicker than 4.5”.
Note: Depending on the number of users or units, these numbers may or may not be enough to act on - it is for you to decide based on your knowledge of your business.
Now that you are beginning to see trends instead of single data points, your data has taken on a higher level of quality; you are moving out of the realm of individual opinion. The more data you collect and theme, the more representative of your population these trends become.
Onward and Upward
Once you begin to see trends emerge, you can start to act on the themes. The benefit of this approach is that you never lose the original inputs – so as you begin to consider solutions with the development team, you have a clear view of the original problems that users need solving. As you develop possible solutions, you can regularly go back and compare them to the original scenario your customers experienced, and potentially even reach out to those specific customers to test prototypes and mockups as you refine them (see our recent post, “Decision-Driven Stakeholder Management” for more about this as well).
Another benefit of this approach is that it enables you to defend against those biases we mentioned earlier. Problems with very little data, for example, should not be prioritized just because an executive had the idea.
Overall, practicing evidence-based product management allows you and your team to make better decisions; decisions based on the accumulation of actual customer data that tell you how pervasive and urgent a problem really is. It helps you to spot themes beyond the loudest or more recent voices, and gives you the ability to combat those biases in objective ways.
For further reading, check out these resources:
- "The Case For NOT Listening To Your Customers' Requests" – For suggestions about how to capture customer requests.
- "What Cleaning Cheerios Out of My Toddler’s Car Seat Taught Me About The Customer Journey" - Thoughts on mapping the customer experience throughout their entire journey with your product.
- “Decision-Driven Stakeholder Management” - Once you begin solution development, this method can help keep decision points and their relevant stakeholders connected.
Adam Shulman is a Product Manager with extensive experience in software/hardware systems and a passion for music and audio technology. He currently leads the Installed Systems business at Bose Professional and has been a member of the BPMA since 2016.
