The ROI of Feedback

The product and research teams tasked with developing solutions are often assigned fixed budgets and timelines within which they are expected to achieve their goals. The ROI of FeedbackThese teams must develop products that customers want and need, and their success is measured by whether or not people actually use the solutions they create. In order to make sure the products they build reflect people’s constantly changing needs and wants, these teams will likely want to get feedback from people so they can evaluate pain points and test concepts. 

Measuring return on investment (ROI) of discoveryThis poses a challenge because fixed budgets and timelines may not include discovery. To an overly confident budget holder, it can look like discovery unnecessarily delays and increases the cost of development. This person may think the organization already knows enough, has the right plan in place, and should move at full speed ahead. If you find yourself needing to convince someone like this, you’ll need to communicate the return on investment (ROI) of discovery.

There are three methods for communicating the ROI of getting the feedback you need: rational, observed evidence (otherwise known as “empirical”), and mathematical. Product and research teams that recognize the importance of discovery must also understand which approach or combination of approaches will resonate most with select stakeholders. Once you have determined your approach, you will be able to present options for discovery that will be understood to be fast, cost-effective, reliable, and most importantly, worth dedicating time and budget to.

Rational: Communicate Using Common Sense Agile Research facilitates rational product decisions by validating assumptions

The simplest and most straightforward way to justify investing in discovery is by using common sense. Product and research teams acknowledge that they live in a world full of unknowns.They take a rational approach to their unknowns:

  • There are virtually infinite things that could be built but relatively few things that people actually want, so it’s important to talk to people before starting to build. 
  • There are more things you don’t know than things you do know, and you should validate your assumptions.
  • The world is changing incredibly quickly, so the things you knew a year ago are no longer true today... and vice versa.

Without gathering feedback from the market, the only way to resolve these issues is by guessing. Even if your organization could guess at a rate than to randomly address the list of unknowns above, the likelihood that those guesses will result in you building one of the few possible “right” solutions is low. You are looking for a needle in a haystack. Therefore, it is rational to assume that any development is a waste of resources until proven otherwise.

Conducting discovery is a way to increase confidence that a specific path is the right one. Of course, any step added to the product development process needs to be quick and cost-effective so that the burden of proof doesn’t outweigh the value of that proof. For an overly simplified example, if a feature costs $10,000 and two weeks to build, it wouldn’t make sense to spend $100,000 and three months proving the demand for the feature. However, it would be very rational indeed to add discovery to your next million-dollar decision.

Empirical: Reflecting on Real-Life Experiences Product Managers must rely on empirical evidence to maintain customer-centricity

Seasoned product and research leaders may choose to relate their real-life experiences to justify the investment in discovery. In every industry (and in virtually every company!), there is a traumatic and recent memory of wasting time and resources building a solution that failed. While there are a number of factors that contribute to a product failing, a lack of market demand is usually one of the top two or three reasons. Using this type of real-life example is known as taking an empirical approach.

It doesn’t matter how smart, famous, and prestigious a brand is: most products fail, and some do so in a way that is both public and humiliating. One need only think of Quibi, Magic Leap, Juicero, and Google+ as examples. In fact, Google alone has had over 200 product failures. Unless you have a Google-sized cash cow that can offset hundreds of costly failures, this is probably a strategy to avoid. 

The most powerful empirical justification for investing in discovery is to point to a recent example within your organization of a product that failed despite a surplus of stakeholder confidence. Such failures are not just costly but embarrassing. They require executives to backtrack on promises to employees and customers, and dramatically reduce trust and confidence in future strategies and initiatives.

So long as it is rapid and cost-effective, discovery cannot fail in a similar fashion. In fact, discovery can substantially help product and research teams to avoid failure by identifying risks prior to development and especially prior to bad press.

Mathematical: Thinking Like the Finance Team Mathematical approach to understanding the ROI of Feedback

Sometimes, common sense and past experience simply aren’t enough to convince a budget holder of the ROI of discovery. In these cases, you’ll need to learn how to think and talk like the finance team. The most sophisticated justification for discovery is the mathematical one that will be most familiar to (and likely resonate the most with) financial analysts.

It’s a multifaceted argument that frames discovery and development as options. We’ll outline the equations momentarily, but the concept is straightforward. In simple terms, think of discovery as a system of low-cost betting. The more bets you place, the more you determine what are--and are not--viable ideas.

The mathematical justification calculates expected value. Expected value is the sum of the return of all scenarios multiplied by their associated probabilities. The simple argument is that conducting some amount of discovery is an incremental cost that increases the overall expected value of an initiative.

Let’s take an overly simplified example for demonstration purposes. Assume the following:

  • Your organization has 20 product ideas to capture a $10M market.
  • 1 of those product ideas would succeed while the other 19 would fail, a reasonable estimation given that 95% of products fail.
  • Developing any of the ideas would cost $600,000 and is one ‘option’ to determine whether the idea will succeed or fail.
  • Conducting discovery on any of the ideas costs $5,000 and is another ‘option’ to determine whether the idea will succeed or fail.

Without conducting discovery, the equation is quite simple. We take the sum of the return of all scenarios: 5% multiplied by the chance of success plus 95% multiplied by the chance of failure and subtract the cost of development.

Figure A. | No discovery
(1/20 * $10M) + (19/20 * $0M) - $600,000 = -$100,000

Our expected value is negative meaning that, even before considering wasted time and opportunity cost, this initiative is expected to lose money for the organization.

Remember though that development isn’t our only “option” for determining whether a product idea will succeed or fail. We can invest in other “options” – some number of rounds of discovery which will change our overall likelihood of success or failure.

Let’s examine what happens when we do discovery and identify bad ideas:

Figure B. | One round of discovery identifies that a single idea would be a failure
(1/19 * $10M) + (18/19 * $0M) - $600,000 - (1 * $5,000) = -$79,000

Figure C. | Two rounds of discovery identifies that two ideas would be a failure
(1/18 * $10M) + (17/18 * $0M) - $600,000 - (2 * $5,000) = -$54,000

As you conduct more discovery, you increase your likelihood of success and therefore your expected value. The breakeven point in the example above is between three and four rounds of discovery (see Appendix for the quadratic formula), but each finance team may use a different calculation taking into account factors like time, opportunity cost, historical track record, and so on.


The Cost of Pivoting

The above is sufficient mathematical proof of the value of options. However, when it comes to digital product development, the reality is a bit more nuanced. Consider that the cost of making significant changes to a product (“pivoting”) increases as you build the product. Therefore the value of doing discovery decreases over time. To look at it from the other direction: feedback is worth more at the prototype phase than at the QA phase.

Understanding the cost of pivoting in product development

If product and research teams are going to invest in discovery, the ROI is much greater the earlier they’re able to evaluate decisions.

Discovery is Convex

All the arguments for discovery boil down to a single reality: discovery is convex. When plotted, the value of discovery bends positively away from the x-axis. 

Customer centricity can be valued as worthwhile insurance for product development

In plain English this means that discovery is an option with an associated cost. There is a possibility that discovery merely confirms what your organization already planned to build. In such a case, discovery can be valued as worthwhile insurance rather than a waste of time and money.

On the left side of the graph above, we see a visualization of this insurance value of discovery. However, as we chart the convex movement of the graph, we clearly see that discovery is not only valuable for preventing you from building disastrous solutions. 

There is a significant potential upside to the investment in discovery. The likelihood of gain increases with the use of discovery, which can expose game-changing, uncapped market opportunities, or help you avoid building costly products and features no one wants. That’s the beauty of convexity: the potential upside (revealing new opportunities and avoiding costly mistakes) more than justifies the known and limited downside (the cost of performing discovery). Discovery is one of the most convex activities in business.

Summing it All Up

Fast, Easy, Reliable discovery

Once key stakeholders are on board with investing in discovery, research, and getting feedback, the question shifts from “should we do it?” to “how should we do it and how much of it should we do?” These are the serious questions companies need to evaluate as they adapt to a rapidly changing world. Whatever justification is used, remember that the faster, more cost-effective, and more reliable the discovery, the higher its ROI and the more of it you can and should invest in.

***

Appendix

The break-even analysis for the example in the article’s scenario can be calculated below:

Break-even analysis for discovery

...meaning that we have to conduct at least 4 rounds of discovery to achieve a positive expected value.

Again, this is an overly simplified example merely to demonstrate that discovery increases the expected value of an initiative primarily by increasing the likelihood of a successful outcome.

The example above only works in a vacuum and not in the real world for some of the following reasons:

  • Obviously you can’t pre-determine that some set of ideas has any specified number of successes or failures. Because of this, unlike in our example above, there’s no guarantee that an unlimited amount of discovery will identify a successful idea (or all failing ideas).
  • Presumably, you would stop doing discovery as soon as you identified the winning idea, not simply when you achieve breakeven. Further, if you discovered the winning idea, you would develop it, which doesn’t count as an ‘additional’ option as in my formula.
  • Beyond the cost of conducting discovery, discovery also requires time, which in a fast-moving market can be costly in its own right.

That being said, your financial analysts will still appreciate the algebraic argument and may even use their own formula to ascertain the impact that discovery has on expected value.

New call-to-action

Subscribe to the Feedback Loop