Home » Uncategorized » Measuring The Black Box

Measuring The Black Box

How to design and implement innovation metrics

More than two decades ago, management guru Tom Peters penned an editorial titled “What Gets Measured Gets Done.” Indeed, one of the findings from the research that Peters summarized in the 1982 business classic In Search of Excellence is that excellent firms use measurements and metrics to make sure people spend time on the things that really matter. 

The theory is simple. A senior manager hoping to influence behavior has no stronger lever than his or her choice of measures. Measures serve as tangible guideposts that help middle and junior managers make the critical on-the-ground resource-allocation decisions that-more than any senior management fiat- ultimately determine a company’s innovation strategy. 

The challenge for companies seeking to improve their ability to create growth through innovation is that the metrics that many companies use to measure innovation run a high risk of actually leading companies in the wrong direction. Managers hoping to unleash their innovative potential need to be mindful of critical measurement traps, think about creating a widespread set of metrics, and ensure their executive dashboard constantly monitors the innovation metrics that matter most. 

Measurement Traps

Putting metrics on innovation is difficult because innovation is a complicated, diffuse activity. Even a metric that seems to make all the sense in the world can actually lead to behavior antithetical to the long-term pursuit of profitable growth. Generally, there are three measurement traps companies should guard against: 

1. Too short a list of metrics. Many companies fixate on a single innovation metric. Some try to calculate the return on their innovation activities. While this metric can be quite useful, on its own it can lead companies to inadvertently prioritize measurable markets over difficult-to measure but higher potential markets. 

No one magic metric measures the right target and aligns incentives appropriately. This is due to the fact that companies that are good at innovation master the ability to introduce different types of innovation. They also recognize that getting good innovation outputs requires tracking the right inputs and the right processes. Having single-minded metrics can lead to prioritizing the wrong things. 

2. Encouraging sustaining behavior. Many metrics implicitly-or explicitly-encourage companies to focus on close-to-the-core sustaining innovations that at best promise incremental returns. These incremental innovations aren’t bad, but they are insufficient for companies seeking to create substantial growth. 

For example, companies that focus on the percentage of revenues from recently launched products have to carefully watch to ensure that they don’t subtly encourage very close-to the- core, low-risk innovations. They can implicitly crowd out the disruptive innovations that transform existing categories and create new ones through simplicity, convenience, accessibility or affordability. 

3. Focusing on inputs over outputs. The goal of any company’s innovation efforts is creating growth. Companies that only track input related metrics run the risk of having resources (particularly scientific ones) work on interesting but ultimately low-impact projects. 

As a simple example of the limits of focusing on input-related measures, consider a 2006 study that highlighted private sector companies with the largest R&D budgets. Leading the pack in the U.S. was Ford, which, advertising notwithstanding, isn’t on anyone’s short list of innovative companies. 

Remember, there is a marked difference between invention and innovation. In They Made America, Harold Evans describes how Thomas Edison would implore his scientists: “We’ve got to come up with something. We can’t be like those German professors who spend their whole lives studying the fuzz on a bee.” Put simply, the output matters. 

Winning Metrics

Organizations like the Boston Consulting Group suggest using a balanced mix of metrics to assess a company’s innovation-related activities. We agree. The metrics described below come in three categories: input focused metrics, process-focused metrics and output-focused metrics. 

Metrics are rarely one-size-fits-all. The best set of measures varies considerably depending on the company, its values, its industry and its aspirations. For example, as part of a project with the American Press Institute, we developed very specific metrics for newspaper companies seeking to transform themselves. The metrics included industry-specific factors, such as the percentage of revenues deriving from new online advertising models, and factors that tied directly to the inhibitors to innovation we found at many newspaper companies. 

Finally, the 15 metrics below are intended to encourage both sustaining innovations that extend the core business and disruptive innovations that build new growth businesses. 

Input-Related Measures 

Financial resources dedicated to innovation. Although in isolation this variable can be dangerous, innovation requires real resource commitment. However, apportioning a huge budget to innovation can lead to the “big bet” trap. In fact, limiting funding can be the right thing to do. Scarce resources force teams to zero in quickly on critical assumptions, find cheap ways to test those assumptions and develop lean, flexible structures. So start with “just enough” and add more.

Resources focused on innovation. This metric ensures that there is dedicated time for innovation activities. In many companies, the really scarce resource isn’t money, it’s time.

Ring-fenced resources for noncore innovations. The previous two metrics ensure that the company generally allocates resources to innovation. It’s important that some of those resources are specifically focused on non-core innovations, and that those resources are fiercely protected, even when times get bad. Kennametal, a $2.3 billion manufacturer of metal cutting tools and mining equipment, established a centralized breakthrough technology group to focus solely on long-term innovations. The group evaluates new technologies, new markets and ways the company can bring significant game-changing innovations to existing markets.

Senior management time invested in new growth innovation. Innovations that are markedly different from core initiatives require careful nurturing from senior leaders. For example, when Procter & Gamble is working on category-creating products, senior executives join in consumer observation sessions, go into the labs to review early prototypes and actively participate in day-long brainstorming sessions.

Number of patents filed. Again, this measure (or an equivalent for non technology companies) can be quite meaningless on its own. But combined with the other metrics, it can be an important interim measure that ensures a constant effort to develop new technology. 

Process- and Oversight – Related Metrics

Process speed. An ideal innovation process quickly moves ideas from conception to critical decision points. That decision point might not always be market launch, it might be a decision to kill or enter a trial market.

Breadth of idea-generation process. Senior management does not have an exclusive license to develop good ideas. In fact, the best ideas can originate from people who are close to markets, such as sales representatives. A good idea-generation process seeks ideas far and wide-from customers, channel partners, even competitors. Measuring the percent of ideas that come from outside the company is a good proxy for the breadth of the idea-generation process.

Innovation portfolio balance. A good innovation portfolio is a balanced innovation portfolio. Balance can exist along multiple dimensions, such as the stage of development, the target domain and the amount of risk. Clorox ensures that investments are balanced in diverse areas ranging from introducing line extensions to creating new categories by classifying its projects into three categories (sustaining, breakout and disruptive) and investing accordingly.

Growth gap. To develop a balanced portfolio, companies should have a good understanding of the gap between their strategic objectives and their current innovation activities. The results must be reasonably risk-adjusted; if success requires that every innovation project meets its current projections, a company should think about developing more (or different) projects. 

Distinct processes, tools and metrics for different types of opportunities. Ideas can look different through different lenses. Tools that help screen and shape core initiatives can unintentionally weed out great- but different-ideas. A company’s core stage-gate process can ruthlessly reshape even the most novel idea to resemble what a company has done before. This metric ensures that a company has different screens, tools and metrics for different types of innovation. For example, IBM classifies opportunities by time-to-market and risk-level and applies the appropriate innovation processes to best cultivate the project. 

Output-Related Metrics

Number of new products or services launched. Clearly, a well oiled innovation machine should produce tangible output. Tracking the number of outputs makes sure the engine is running appropriately.

Percent of revenues in core categories from new products. As mentioned above, this metric in isolation can unintentionally encourage needless line extensions. But coupled with other metrics, it can ensure that a company has appropriately seized the close-to-the-core opportunities that are critical to growth.

Percent of profits from new customers (or occasions). New growth innovations should create legitimately new growth. Why focus on profits? An important innovation lever is the business model. Focusing on profits gives innovators the freedom to tinker with the profit formula, charging lower prices but increasing velocity, or actually charging higher prices and earning more attractive margins.

Percent of profits from new categories. Not only should innovative companies be able to reach new customers or new usage occasions; they should be able to create (or play in) entirely new categories that didn’t exist several years ago. This metric forces innovators to look for innovative opportunities, remembering that most great growth businesses will start a step or two away from the core. Kellogg’s Special K brand measures its profits from new categories such as branded snack foods and beverages, forcing the brand team to continually look for new opportunities.

Return on innovation investment. ROI can again be a dangerous metric in isolation, forcing innovators  to prioritize sure bets that promise modest-at-best returns over riskier but  potentially more lucrative propositions (Net Present Value, which doesn’t suffer from this sizing problem has its  own complement of issues). Nonetheless, companies shouldn’t fritter away innovation resources on activities that don’t demonstrate returns. 

From Theory to Practice 

Implementing innovation metrics is no easy feat. The following steps can ease the process.

1. Focus, focus, focus.  In Animal Farm, George Orwell wrote, “All animals are equal, but some are more equal than others.” The same is true of innovation metrics. All the metrics described above matter, but some matter more. Furthermore, what matters depends sharply on a company’s circumstances, capabilities and strategic objectives.

Start by coming to consensus round your innovation strategy and zero in on the company-specific barriers that inhibit your ability to create growth through innovation. Attempting to calculate the growth gap, while difficult, can be a very useful input into this process.

No one manager should feel like they are on the hook for delivering against a dozen different metrics. The list of metrics for any one manager needs to be limited so as to provide a mental map that supports the right behavior. Too many metrics can lead managers to prioritize what they think is important instead of what really matters. 

2. Remember relativity. A company can look great along each of the metrics but still find itself falling behind competitors. It is always important when using metrics to analyze not just internal progress, but progress against external benchmarks. Although many of the metrics described in this article require internal knowledge that makes external tracking difficult, it’s still worth the effort. 

3. Innovate the innovation metrics. Any company that installs a batch of metrics needs to make sure that it is willing to constantly update those metrics. Often, the right metrics are only available in hindsight, so be ready to add, drop and change any metric you adopt. 

4. Gain alignment up and down the corporate chain. It does no good for a business unit within a conglomerate to adopt metrics that differ markedly from those of the conglomerate’s CFO. The unit that tries to push in new directions will find itself pulled back to the corporate path.

Following these steps can help companies implement a series of innovation metrics that bring greater clarity to innovation efforts. The right metrics can help align managers in ways that increase a company’s odds of developing a robust innovation portfolio.


Scott D. Anthony is president of Innosight, an innovation consulting and executive training firm based in Watertown, Mass. Steve Wunker is a partner and Steven Fransblow is a senior associate at Innosight.

About scott d. anthony steven fransblow and steve wunker